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Introduction 


This report aims to satisfy Air Traffic Management Technology Demonstration — 1 
(ATD-1) Statement of Work (SOW) 3.6.19 and serves as the delivery mechanism for the 
analysis described in Annex C of the Flight Test Plan. The report describes the data 
collected and derived as well as the analysis methodology and associated results extracted 
from the data set collected during the ATD-1 Flight Test. 


All analyses described in the SOW were performed and are covered in this report except 
for the analysis of Final Approach Speed and its effect on performance. This analysis was 
de-prioritized and, at the time of this report, is not considered feasible in the schedule and 
costs remaining. 
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1. Data Set 


The data set described in this document was obtained during the execution of the flight 
test program described in the Flight Test Plan document (D4.22). The data set consists of 
Electronic Flight Bag (EFB) data collected on the two aircraft performing FIM during the 
flight tests conducted Jan 20-Feb 22 2017 in the Seattle, WA, area. Aircraft state data was 
collected from all three aircraft in the flight test. The aircraft were a United 737-900, a 
Honeywell 757, and a Honeywell Dassault F900. Only the 737 and the 757 are equipped 
with Flight Deck Interval Management (FIM) and flew FIM missions. The F900 acted as 
a lead in a merging chain of aircraft on all but two flight test days. No significant 
difference in performance has been observed because different lead aircraft are used. 


Nineteen flying days resulted in 198 unique FIM operations. Approximately 11% of the 
experiments were discarded because of bad or unusable data that resulted from operator 
error, software errors, and scenario setup difficulties. Of the good data, 80% was retained 
for performance and other analysis included in this report; the remainder (9%) was 
flagged for further study to investigate system behavior deemed incorrect or not fully 
understood. The data set used for this analysis report consists of 144 flown FIM 
experiments (runs): 11 en route (A conditions) experiments, 8 final approach spacing 
experiments, and 125 arrival/descent experiments (B conditions). The data set includes 
75 conditions for the 757 and 69 conditions for the 737. Of the four clearance types the 
prototype FIM system can handle, the data set contains 25 MAINTAIN, 36 CAPTURE, 
75 CROSS, and 8 SPACE operations. Fifteen of the operations used a distance-based 
spacing goal and 129 used a time-based spacing goal. 


The Flight Test Data was delivered to NASA under Deliverable 4.24 on March 29, 2017. 
An accompanying document on the delivered physical media describes the form of the 
data, and Appendix C of the Flight Test Plan describes its content, with differences 
described in the next section. 


Differences from Annex C of Flight Test Plan 


Many more data items are found in the delivered Flight Test Data than are documented in 
Annex C of the Flight Test Plan. Some of the mappings between data items in some logs 
and the NASA data requests are found in the “NASA ATD1 Data Recording details_v4- 
4.xlsx” spreadsheet accompanying D4.24. The major difference between what was 
collected and what was planned to be collected as documented in Annex C of the Flight 
Test Plan relates to the Honeywell 757 aircraft and the collection of Flight Management 
System (FMS) data. 


The Cross-Track Distance (ARINC Label 116) and Vertical Deviation (ARINC Label 
117) parameters from the FMS are not valid in the 757 aircraft state data recordings. 
These data were previously reported to be available based on published Interface Control 
Document (ICD) data; however, examination of the data during flight test shows only 
zeroes were recorded. Unfortunately, spot checking of the data recordings made after 
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equipment installation and periodically during the flight test did not reveal these data 
were invalid until very late in the flight test campaign. At this time, it is not known if this 
results from FMS configuration or FMS software version differences between the 
published ICD and the installed software. 


Consequently, the FMS Flight Technical Error (FTE) metrics could not be extracted for 
the 757 and thus the associated derivation of FIM system Path Definition Error (PDE) 
could not be performed for that aircraft (see IM Performance Metrics section for a 
description). 
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2. Individual Run Graphical Data 


For each run, several graphical artifacts are produced to inspect its general health and 
performance. Many of these artifacts are used during data review to identify which runs 
to keep based on an initial assessment of performance. The major types of graphical 
artifacts that can be produced from the Matlab code developed under the program is 
described later in this report, including a few examples for a number of runs. This code is 
not itself a deliverable on the program but is shared with NASA as a courtesy. 


Interval Management Performance 


Interval Management (IM) performance is depicted in the form of a Measured Spacing 
Interval/Predicted Spacing Interval (MSI/PSI) versus a distance-to-go (DTG) chart with 
many overlaid pieces of information. The alignment of all data is accomplished through 
the common Coordinated Universal Time (UTC) time stamp present in all FIM logs and 
non-FIM data logs. An example of such a plot for condition B19 (CROSS) on Day 9 is 
shown in Figure 1. 


The chart is bounded by IM operation start on the right (green vertical line), and ownship 
passing PTP on the left (DTG=0 nm). The black horizontal line near the middle is the 
assigned spacing goal (ASG) used for the operation. The blue line (or set of dots) are the 
MSI or PSI reports in the units of the operation (time or distance). Each IM speed issued 
is represented by a magenta circle, with the value and UTC time of the issued speed as 
accompanying text. Trajectory generation events are shown as squares with different 
colors for ownship and traffic. Vertical lines of different styles are depicting key 
times/events, such as ownship and traffic passing Achieve-by Point (ABP) or PTP. This 
type of chart is typically inspected to get a sense of the evolution of the spacing error and 
associated issued IM speeds (values and frequencies) and Trajectory Generator (TG) 
generation events. For operations that involve the use of two different spacing algorithms 
in the prototype (Constant Time Delay [CTD] and Trajectory-Based Operations [TBO]), 
both MSI and PSI are shown jointly on the one graph (Figure 1) with PSI reported by the 
TBO algorithm up to the point ownship passes the ABP. MSI is then reported by the 
CTD algorithm up to the point ownship passes the PTP. 
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Figure 1: IM Performance Plot Example, Day 9, B19 


IM Speed Performance 


IM Speed Performance is depicted in the form of calibrated airspeed (CAS) versus DTG 
with many overlaid pieces of information aligned using common UTC time stamps. The 
blue line represents the nominal speed profile generated by the Trajectory Generator 
(TG). The green line represents the actual flown CAS by ownship. The magenta line 
represents the IM commanded speeds issued, and the red line represents the instantaneous 
commanded speed. Each trajectory generation event is depicted by a square; vertical lines 
indicate key times as in the IM Performance chart. An example of such a plot for the B19 


condition flown by the 737 on Day 9 is shown in Figure 2. 
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IM Speed Plot - Ownship: Run B737, B19, Day9 
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Figure 2: IM Speed Plot Example, Day 9, B19 


Figure 2 reveals four instances where pilots dialed Mode Control Panel (MCP) speeds 
that did not correspond to an issued IM speed prior to NALTE. This is atypical and can 
be caused by pilots either choosing to not follow the FIM speed at their discretion (safe 
speeds, turbulence penetration limits, etc.) or follow Mach issued speeds, which for the 
figure are translated to CAS. 


IM Speed Limiting Information 


The IM speed limits applied by each algorithm during an IM operation is depicted along 
DTG. Figure 3 and Figure 4 depict examples of such plots for condition B19 on Day 9. 
Because this is a CROSS operation, the first example shows the limiting applied during 
the achieve portion of the operation by the TBO algorithm. Each dashed line represents 
an applicable limit. The thin green line is the ownship CAS and the magenta line is the 
IM speed issued by the algorithm. If the IM speed is limited, the line is thickened during 
the limiting portion and is sitting on top of one of the limiting lines. For example, in the 
condition depicted, the maintain portion of the operation (CTD algorithm) was limited on 
the low side in the last 5 miles (Figure 4). If the IM speed line sits on top of a limit and is 
not thickened (e.g., 40-37 nm DTG in Figure 4) the system is not technically reporting a 
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CAS (kts) 


limit condition and no limiting indication is sent to the pilot through EFB or CGD. It only 
indicates a limit condition when the algorithm desires a higher or lower speed but cannot 
issue it because of the limit. 
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Figure 3: TBO Speed Limiting Example, B19, Day 9 
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Limit data - CTD 


——NomProfile Upper 
~——— Nomprofile Lower 
——VMO 


—— MMO (as CAS) 
~~~ IM Upper CAS 


—— IM Lower 


—— OwnshipCAS 
——IM CAS 


~ IM Upper Mach (as CAS) 


30 40 50 
Ownship DTG to PTP (nm) 


Figure 4: CTD Speed Limiting example, B19, Day 9 


Lateral TG 


The lateral view of the flown experiment overlays the actual lat/lon state of ownship on 
top of the generated trajectory. There will be one such plot for every trajectory 
generation, both for ownship and traffic. By default, the plotting only occurs for 
generations occurring at IM start and beyond. Key points in the procedure (waypoints, 
ABP, PTP) are shown in Figure 5, as well as all lateral and vertical Trajectory Change 
Points (TCP) (magenta circles). 


The along-track wind forecast error (ATWFE) is also overlaid on top of the geometry to 
get a sense of its magnitude and sign (headwind or tailwind forecast error) throughout the 
IM operation. Figure 5 displays an example of such a plot for Day 9, condition B19, for 
ownship aircraft being the 737, following the 757. Figure 6 is the companion figure for 
its lead (757). Figure 5 shows that the 737 was subject to a large (~25 kt) forecast 
tailwind error in the upper part of the operation, reversing sign in the middle, with near 
zero forecast error in the low altitude portions. Almost all operations through the RF turn 
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60 


Latitude (deg) 


in the SUBDY approach to runway 32R (RW32R) experience a change in sign and 
magnitude in the forecast error, likely caused by significant changes in altitude and track 
direction through that segment. 
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Figure 5: Lateral Ownship (737) TG Plot Example: Day 9, B19 
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Lateral Plot - Traffic: Run B737, B19, Day:9, GenTime 22:30:09, GenReason INITIAL_TRAJECTORY ATWFE (kts) 
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Figure 6: Lateral Traffic (757) TG Plot Example: Day 9, B19 


Vertical TG 


The vertical view of the flown experiment overlays the actual altitude of ownship and 
traffic on top of the generated vertical trajectory against DTG. There is one such plot for 
every trajectory generation, both for ownship and traffic. By default, the plotting only 
occurs for generations occurring at IM start and beyond. Key points in the procedure 
(waypoints, ABP, PTP) are shown in Figure 7, as well as all lateral and vertical TCPs 
(magenta circles). Constrained waypoints are indicated in Figure 7 with a square (AT), 
upright triangle (AT or ABOVE), upside down triangle (AT or BELOW), or a vertical 
line segment (WINDOW). Figure 7 exemplifies such a plot for ownship on condition B19 
of Day 9 and Figure 8 for the lead aircraft of that pair. 
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4 Vertical Plot - Ownship: Run B737, B19, Day:9, GenTime 22:30:09, GenReason INITIAL_TRAJECTORY 
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Figure 7: Vertical TG (Ownship) Plot Example, Day 9, B19 
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4 Vertical Plot - Traffic: Run B737, B19, Day:9, GenTime 22:30:09, GenReason INITIAL_TRAJECTORY 
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Figure 8: Vertical TG (traffic) example, Day 9, B19 


Speed Profile TG 


The speed profile view of the flown experiment overlays the actual CAS of ownship and 
traffic on top of the generated nominal speed profile in the trajectory against DTG. There 
is one such plot for every trajectory generation, both for ownship and traffic. By default, 
the plotting only occurs for generations occurring at IM start and beyond. Key points in 
the procedure (waypoints, ABP, PTP), as well as all lateral and vertical TCPs (magenta 
circles), are shown in the figure. Constrained waypoints are indicated in the figure with 
an upright triangle. Unconstrained waypoints are depicted at the 300 kt CAS level with a 
star as marker. The blue line in the figure is the generated nominal speed profile; the 
green line on the ownship figure is the CAS found in the ownship state IM log, and the 
red line on the traffic figure is the CAS found in the time-correlated aircraft state data. 


Note that the plot does not include IM speed information--only the nominal speed profile 
and the flown CAS--which may reflect pilots following issued IM speeds in cases where 
the plot is for a FIM aircraft (Figure 9). In the case of the plot for the pair lead (traffic) 
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(Figure 10), the flown CAS (red line) may also reflect IM speeds followed by that aircraft 
when it was performing FIM (it is in the example). 


In cases where the lead aircraft is not performing FIM (lead of pair 1, Figure 11), the 
figure reflects the delay condition flown by the lead. Such plots were used during flight 
test to stress the importance of achieving a given lead aircraft delay condition repeatedly 
in cases where it could be demonstrated that pilots did not fly the intended speed profile. 
The example shown in Figure 11 is for the first pair for condition B19 on Day 9, where 
the lead aircraft is the F900 flying relatively close to the nominal CAS profile for a no- 
delay scenario. 
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Figure 9: TG Speed Profile Example (Ownship), B19, Day 9 
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Speed Plot - Traffic: Run B737, B19, Day:9, GenTime 22:30:09, GenReason INITIAL_TRAJECTORY 
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Figure 10: TG Speed Profile Example (Traffic), B19, Day 9 
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Speed Plot - Traffic: Run B757, B19, Day:9, GenTime 22:26:47, GenReason INITIAL_TRAJECTORY 
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3. IM Performance Metrics 


Each run was analyzed to extract a set of 34 metrics capturing IM and aircraft 
performance. Each metric is defined here, including its method of computation. Some 
data related to the description of the run are described as well, although they are 
technically not metrics used for analysis. The IM Performance data was delivered under 
Deliverable 4.25a on March 29, 2017. An accompanying document on the delivered 
physical media describes the form of the data. Description of the data from that document 
is replicated in the following sections. 


IM Performance Data Description 


Each run was first inspected to determine the bounds of the IM operation (start and end 
times). All computations involving determination of the actual time of arrival (ATA) at a 
particular point are based on computing the closest time of approach to that point (based 
on position definition in the flight plan EFB log) using a spline fitting process to the set 
of great circle distance from the time history aircraft position data and the fixed point. 
The spline is fitted through the actual data for time history position points within a 
threshold distance, typically 0.5 to 1.0 nm. 


The IM performance data is tabulated in an Excel spreadsheet for each of the 144 runs 
that are part of this analysis. The raw and derived data is in Matlab form and exists in a 
corresponding MAT file, both as a structure array (“ATD1_metrics_array”) and a Table 
Matlab variable (“ATD1_ metrics table”). The following paragraphs describe each metric 
and how it is computed. The metric name (in italic) matches the column name in the 
Excel spreadsheet. The Matlab structure array field name and corresponding Table 
variable name are also included in the description. 


The start time (tStart) in most cases is automatically determined from the data based on 
the time the system went to the PAIRED state, at the ASG for the operation. In cases 
where data were lost or significant suspend events (more than a few seconds) exist 
downstream of the automatically determined start time, the start time was manually 
adjusted by inspecting the time history plot of MSI or PSI. 


The end time (tEnd) is defined per the FIM Minimum Operational Performance Standards 
(MOPS) as 


1. Time when traffic passes the PTP (distance-based operations). 

2. Time when ownship passes the PTP (time-based operations). 
Once the start and end times are known, the IM operation duration ([Mduration) and 
distance ([Mlength) are computed. [Mlength is the along-track distance covered by 


ownship between tStart and tEnd. /Mduration is the difference in time between tEnd and 
tStart. 
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The Initial Spacing Error is computed as the difference between the MSI (or PSI) at 
tStart and the ASG. Its units are in seconds or nautical miles, based on the units of the 
operation (time or distance). Based on this definition, a positive value indicates the IM 
operation aims for the aircraft to speed up (decrease spacing); a negative value aims for 
the aircraft to slow down (increase spacing). 


The initial ABP distance (DTG_ABP) is computed as the ownship along-track distance 
remaining to the ABP at tStart. This is a computable metric for CROSS operations only. 


The initial ABP predicted time (TTG_ABP_pred) is the reported ownship time to go 
(TTG) to the ABP at Start. This is a computable metric for CROSS operations only. 


The achieved spacing error at the PTP (achievedSpacingError_PTP) is defined as 


1. The difference between ownship and traffic ATA at the PTP and the ASG, for 
time-based clearances. 


2. The difference between the along-track distance remaining for ownship when 
traffic reaches the PTP and the ASG, for distance-based clearances. 


The achieved spacing error at the ABP (achievedSpacingError_ABP)) is defined as 


1. The difference between ownship and traffic ATA at the ABP and the ASG for 
time-based clearances. 


2. The difference between the along-track distance remaining for ownship when 
traffic reaches the PTP and the ASG, for distance-based clearances. 


The number of times the system was speed limited (numLimited) is a count of the speed 
limited reports in each of the logs (CTD and TBO). Because the logs are at 1Hz, it is 
equivalent to the number of seconds (not necessarily consecutively) the system was speed 
limited. The details of which speed limit condition is applied are found in the Matlab 
structure array data. The field “limitData” contains a 2 by 1 cell array with each cell 
element being a Map with the key being the limit condition and value being the number 
of times the algorithm reported that limited condition in the log. The first cell is for the 
TBO algorithm and the second cell is for the CTD algorithm. 


The number of IM speeds (numSpeeds) is a count of unique speeds issued by the IM 
system, including whatever speed is displayed by the system at tStart. This includes both 
algorithms and both Mach and CAS values. If the speed remains the same across an 
algorithm transition (TBO to CTD), it is not counted separately. For distance-based 
operations, speeds issued after the lead passes the PTP are counted separately (see 
numSpeeds GS_ segment). Speeds are included in the Matlab structure array data in the 
field “speeds” in order of issue. 


The number of IM speeds per minute (numSpeeds_perMin) 1s the ratio of numSpeeds to 
IMduration. 
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The number of IM speeds in Ground Speed matching mode (numSpeeds_GS_segment) is 
defined as the number of IM speeds (CAS or Mach) displayed to the pilot by the system 
in distance-based operations after the traffic reaches the PTP. During the interval between 
traffic and ownship passing the PTP in distance-based operations the IM system does not 
maintain the ASG, it only attempts to have ownship pass the PTP at the same ground 
speed as its lead aircraft passed the PTP. 


The number of speed reversals (numSpeedReversals) is a count of the number of times an 
IM speed issued has changed direction from the previous speed issuance (going down 
after having gone up and vice versa). This does not include reversals that may occur in 
the ground speed (GS) segment. 


The pilot time metrics (pilotTime_mean and pilotTime_std) are computed for each run by 
cross correlating the FIM and non-FIM data through the UTC time, which is present in 
both logs. For each IM speed issued, in order, a corresponding MCP value is searched for 
starting at the time of IM speed issue forward to the minimum of 5 minutes or the time of 
the next IM speed issued. If a value within 1 kt (for CAS) or 0.01Mach (M) is found, the 
time difference between the IM speed issued and that time is reported as the time (in 
seconds) the pilot took to implement the IM speed in the MCP. Ifa value is not found, a 
NaN value is reported. The mean and standard deviation of these times for each 
experiment is contained in the corresponding variables. Note that the NaNs do not affect 
those statistics, and thus the reported pilot times are not affected by times when pilots 
chose not to implement a displayed IM speed or missed it altogether. The details of each 
IM speed issued and its associated pilot implementation is found in the Matlab data. The 
data structure fields pilotTime_cas and pilotTime_mach contain arrays where the first 
column is each IM speed and the second column is the elapsed time between display and 
pilot action in seconds. 


The time of capture (tCapture) is defined as the elapsed time (in seconds) between IM 
operation start and the transition between the capture and maintain segment, for 
CAPTURE operations only. The point of capture is defined as the point in time when the 
error is first within +10 s of the ASG. For distance-based operations, the time-based error 
is used (it is found in the logs). 


The maintain-stage error (spacingErrorMaintainSegment_rms) is defined as the root 
mean square (RMS) of the time-based history of the difference between the MSI and the 
ASG for the maintain portion of the IM operation. This portion is the entirety of a 
MAINTAIN operation, the portion downstream of the capture portion of a CAPTURE 
operation (see tCapture definition previously) or the portion downstream of the ABP of a 
CROSS operation. This is in units of the ASG. 


The MOPS maintain metric (MOPS_maintain_metric) is the MOPS defined metric for 
the maintain stage of IM operations. It is defined as a percentage of the maintain segment 


time where the error was outside of 10 seconds from the ASG. 


The along track wind forecast error (ATWFE) is defined as the along-track component of 
the difference between the wind forecast and the sensed wind. Sensed wind (speed and 


REV A D780-10416-1 Page 23 of 116 


direction) is available in the EFB data set at approximately 1 Hz. The forecast wind 
(speed and direction) is interpolated based on the wind forecast entry and the aircraft 
altitude, which are available at approximately 1Hz. Thus, the wind forecast error can be 
computed at 1 Hz and projected along the aircraft ground speed and track at the same 
frequency. The complete time history of the ATWFE between IM operation start and end 
times for both ownship and traffic is computed. The aggregate metric per operation is 
computed as the RMS of that error. The difference in the error between traffic and 
ownship is also computed and extracted as an RMS value for each run. The raw values 
for ATWFE for ownship, traffic and their difference are in the Matlab data under 
structure field names ATWE, ATWE_traffic, and ATWE_diff correspondingly. 


The fuel burn metric (fuelBurn) is defined as the fuel expensed by ownship between the 
IM operation start and end times in pounds. 


The average fuel rate (avg_fuel_rate) is computed as the ratio of fuelBurn to [Mduration, 
reported in lbs/hr. It corresponds to the average fuel burn rate over the IM operation. 


The lateral path FMS deviation (latDev_FMS_RMS) is reported by the FMS as the 
difference between the aircraft position of the FMS reference lateral guidance path. The 
RMS value of this error history is the aggregate metric. Because the 757 did not produce 
FMS deviation data, this metric is computed only for the 737. 


The vertical path FMS deviation (latDev_FMS_RMS) is reported by the FMS as the 
difference between the aircraft altitude of the FMS reference vertical guidance path. The 
RMS value of this error history is the aggregate metric. Because the 757 did not produce 
FMS deviation data, this metric is computed only for the 737. 


The IM system computes the deviations to its computed trajectory (lateral and vertical) 
for both ownship and traffic. The RMS values for both lateral and vertical deviations, for 
both ownship and traffic are correspondingly reported as aggregate metrics as 
latDev_FIM_ownship_RMS, vertDev_FIM_ownship_RMS, latDev_FIM_traffic_RMS, 
vertDev_FIM_ownship_RMS. Lateral deviations are reported in nm, vertical deviations in 
feet. The difference between the FMS- and FIM-reported deviations corresponds to the 
difference between the FMS computed guidance path and the FIM constructed trajectory, 
a measure of instantaneous FIM System Path Definition Error (PDE) as described in 
Annex C of the Flight Test Plan. The RMS values for that difference for both lateral and 
vertical dimensions are reported as PDE_lat_rms and PDE_vert_rms in nm and feet. 
Because the 757 did not produce FMS deviation data, PDE metrics are computed only for 
the 737. 
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4. Summary Statistics 


The summary statistics for each category of IM operation are shown in Figure 12 using 
box plots. Box plots are standardized statistic graphical representations of slices of the 
data set depicting the median, lower and upper quartiles outliers for each metric. The 
values for those descriptive statistics can also be found in the D4.25a Excel spreadsheet 
under the “Descriptive Stats” tab and associated tables in this section under each figure. 
Boxplots are only shown for slices of the experiment matrix with sample size of seven or 
more. Tables also include the 95% confidence interval for each data item from the 
samples. Performance metrics reported include input conditions (initial spacing error) as 
well as IM performance in the form of delivery performance (at PTP and ABP), number 
of IM speeds, IM speed rate, and MCP times. In addition, for CAPTURE operations, the 
time-to-capture is included. For operations with a maintain segment the maintain segment 
RMS error is shown. 


Note that the intent of the contracted effort was not to prove that the prototype 
implementation meets the MOPS performance, but rather is a first attempt to measure 
performance of the first prototype of a FIM system flown in realistic operations. 
Therefore, mentions are made to some, but not all, MOPS performance criteria in this 
report. Most of the criteria investigated are PTP and ABP delivery performance (within 
10 seconds 95% of the time). 


All Time-Based Operations 


The time-based operations statistics are shown in Figure 12. This figure clearly shows the 
power of the IM system, taking the Initial Spacing Error at IM initiation (median=-15.981 
s, IQR=33.57 s) and delivering a much smaller spacing error at the PTP (median=0.9506 
s, IQR=5.36 s). This is done by correcting aircraft speeds on average 10.6 times 
(median=10, IQR=6) or 0.57 speed corrections per minute of operation 

(numSpeeds rate), with the pilot taking an average of 8.51 (MCP time mean) with 
dispersion of 5.8 seconds (mean of MCP time std). 
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Figure 12: IM Performance - All Time-Based Clearances 


Because of the design of the experiment matrix and the difficulty in setting up each 
condition, the initial spacing error is not normally distributed (Figure 13). Many of the 
experiments had near zero initial spacing error, as is expected for MAINTAIN 
operations. However, the resulting spacing error at the PTP is close to a normal 
distribution between -5 s and +5 s of error, which covers ~70% of the data. In fact, 
constructing a cumulative distribution function for the absolute PTP error distribution 
reveals that the probability of the delivered error at the PTP being within 10 s is 89.58% 
(Figure 14). Absence of true normality warrants specialized techniques when performing 
Analysis of Variance (ANOVA) on the data presented in the next section. 
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Figure 13: Initial Spacing Error Distribution (Time-Based Clearances) 
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Figure 14: Cumulative Distribution of Mean Absolute PTP Error (Time-Based Clearances) 


All Distance-Based Operations 


All distance-based operation statistics are shown in Figure 15. The sample size is much 
smaller (15) but the statistics reveal similar trends. The initial spacing error has a median 
of 0 nm (IQR=3.55 nm) as many of the distance-based runs were in fact en route 
MAINTAIN operations. However, the delivered error has less spread (IQR=0.8 nm). The 
number of speeds issued is on average 5.8 (median=4, IQR=5.75), slightly less than time- 
based clearances, although they are likely not statistically comparable because of the 
difference in sample size. 
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Distance Based Metrics (nm) / Pilot Time (s) 


All Distance Based Clearances: 15 
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Figure 15: IM Performance - All Distance-Based Clearances 
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Time-Based MAINTAIN Operations 


Looking at each clearance type operated in time (sample is much larger), we start with 
MAINTAIN operations in Figure 16. Note from Figure 16 that the median of the initial 
spacing error is not zero. This is caused by two main factors: 


1. In MAINTAIN operations, the ASG is set when the system transitions to the 
AVAILABLE state, but the initial error is computed when the system transitions 
to the PAIRED state. Pilots typically wait about a minute in the AVAILABLE 
state to verify system function during which time the system is operated in open- 
loop fashion. Thus, during the two state transitions, the indicated MSI can drift 
and thus a non-zero initial error can occur. 


2. Several runs start time were adjusted to portions downstream of the system 


transition to the PAIRED state for a variety of reasons. This results in the same 
effect as in item 1; however, there is much less frequency of this occurring. 
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Time Based Metrics (s) 


All Time-Based MAINTAIN Clearances: 22 
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Figure 16: IM Performance - Time-Based MAINTAIN 


The statistics show that the PTP delivery error is further from zero than the initial spacing 
error with larger spread. However, no sample of PTP delivery error was greater than +10 
s. The standard deviation of the delivery error is 2.84 s, and assuming the data is 
normally distributed, corresponds to a delivery error of 5.68 s at 95% probability, well 
inside the MOPS requirement. To understand why the delivery error spread at the PTP is 
larger than the initial error, we need to understand the variability in spacing error 
throughout the operation. MAINTAIN operations should also keep the spacing within 
the error tolerance for the entire maintain segment, not just at the PTP. A more 
representative metric of the MAINTAIN operation is the RMS value of the spacing error 
(MSI-ASG) throughout the IM operation. Data show that the system maintained the error 
to within less than 3 s throughout operations ranging from 40 to 110 nm. The 95% 
confidence interval for the RMS maintain error is between 2.05 and 3.35 s. 
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Time Based Metrics (s) 


As noted in many pilot surveys, the number of speeds issued to realize this performance 
is large (median=15.5, IQR=12). In fact, the MAINTAIN operations have the largest 
number of IM speeds issued per minute of all operations flown (0.7 speeds per minute 
mean). This is a product of the state-based algorithm (CTD algorithm) being overly 
reactive to changes in the lead speed, discovered using history, and predicting 
deceleration regions. 


A subset of the MAINTAIN operations (four runs) was performed at cruise level (A 
conditions). Those statistics are contained within the set shown in Figure 17 and do not 
represent outliers. The A runs exhibit less speed changes owing to the fact that the 
operations occur in Mach at level flight and thus there is less opportunity in the CTD 
algorithm deceleration identification to return more data. 


En-Route Time-Based MAINTAIN Clearances: 4 
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Figure 17: IM Performance - En Route Time-Based MAINTAIN 
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Time-Based CAPTURE Operations 


The time-based CAPTURE statistics are shown in Figure 18. Time-to-capture (tCapture) 
in the boxplot is shown in minutes to match the scale of other time-based metrics that are 
in seconds. The delivery statistics at the PTP (median=0.7576 s, IQR=2.59 s) are very 
good relative to the initial spacing error (median=-21.43, IQR=73.83 s). The time-to- 
capture averages at 5.57 minutes (mean), with an IQR of 3.72 minutes. The MOPS metric 
for capture performance is based on a minimum capture rate of 3 seconds of error per 
minute of flight time. The capture rate for CAPTURE operations is computed by dividing 
the initial spacing error (reduced by 10 s) by the capture time in minutes. The mean 
capture rate for CAPTURE operations is 6.1 s, with only 4 out of the 34 CAPTURE 
operations below the 3-s threshold. Three of those four were for cases where the IM 
aircraft was the third in the chain, and all of them were for a delay case for the chain lead 
of at least medium (~10 kt below profile), all challenging cases for the algorithm. 
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Time Based Metrics (s) 


All Time-Based CAPTURE Clearances: 34 
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InitialSpacing IM Length numSpeeds__pilotTime_mean MaintainError 
Error (s) (nm) PTP error(s) numSpeeds perMin (s) pilotTime_std (s) _RMS (s) tCapture (s) 
Mean -6.58 88.35 0.19 10.47 0.54 10.03 8.93 3.91 334.06 
95% Cl low -21.99 80.90 -0.81 8.89 0.48 8.37 5.84 3.29 260.69 
95% Cl high 8.82 95.80 1.18 12.05 0.60 11.69 12.02 4.52 407.43 
Range 127.95 79.75 13.60 17.00 0.69 20.93 32.16 6.91 1047.00 
IQR 73.83 22.60 2.59 7.00 0.25 6.10 8.13 1.81 223.00 


Figure 18: IM Performance - Time-Based CAPTURE 


The number of speeds required to achieve this performance averages 10 (median) with an 
IQR of 7. The maintain segment of the operation has similar performance to equivalent 
MAINTAIN operations shown before (i.e., less than 4 s of RMS error). 


Only two scenarios make up the portion of CAPTURE operations performed en route (A 
scenarios). One of those runs is, in fact, an outlier in Figure 18 (top outlier on PTP error). 
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Time-Based CROSS Operations 


The CROSS statistics are shown in Figure 19. By far the largest number of IM operations 
performed during flight test are time-based CROSS operations. Sixty-eight such runs are 
part of this analysis. CROSS operations are more complex than either MAINTAIN or 
CAPTURE operations as they allow for merging geometries and large portions of the 
operation being on different routes. CROSS operations rely on a predictive algorithm 
rather than a state-based algorithm for the spacing signal on which the control is operated 
(PSI). CROSS operations attempt to achieve the desired spacing at the ABP, which may 
be upstream from the PTP. Between the ABP and PTP, the operation is performed on the 
same algorithm as CAPTURE and MAINTAIN operations, and that segment of the 
operation is labeled the “maintain segment.” The flight test matrix includes many 
combinations of geometries and associated ABP and PTP placements. Some have no 
maintain segment; in particular, those with ABP and PTP at ZAVYO. Many of these 
operations were performed with a merging geometry with the lead on the SUBDY arrival 
and the FIM aircraft on the UPBOB arrival (late merge), but some were also performed 
with two aircraft on SUBDY having a prior merge at NALTE. Some CROSS operations 
with the merge at NALTE were performed with the ABP placed there. 
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Time Based Metrics (s) 


All Time-Based CROSS Clearances: 68 


[ [ [ [ [ [ [ [ 
ae 
80;— = 
60|— 4 
" 
40 a 4 
20}- ae “> a E =] 
eae 
0 ———— ara + 
-20- — =| 
-40 | 
-~60 4 
-80 -— —— 4 
+ 
100 |— | 
I [ I l [ [ [ [ 
Initial error PTP error ABP error MaintaintSegmentErrorRMS Num Speeds NumSpeeds_rate MCP time (rms) MCP time (std) 
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Error (s) (nm) PTP error(s) ABPerror(s) numSpeeds Min (s) (s) RMS (s) 
Mean -17.49 83.27 3.49 3.24 9.78 0.53 8.13 5.08 3.84 
95% Cl low -25.22 79.56 al 7A 1.16 8.91 0.48 7.38 3.45 3.00 
95% Cl high -9.77 86.99 5.27 5.32 10.65 0.57 8.87 6.71 4.68 
Range 180.00 72.49 34.22 38.25 16.00 0.80 16.20 41.05 7.80 
1QR 34.63 18.67 7.82 10.87 5.00 0.22 2.94 2.94 2.82 


Figure 19: IM Performance - Time-Based CROSS 


We first look at the totality of time-based CROSS operations (Figure 19). The delivery 
statistics at the PTP (median=2.0134 s, IQR=7.8206 s) are larger than for CAPTURE 
operations and biased more towards arriving late, but require on average about the same 
number of speed corrections. The standard deviation of PTP delivery error is 7.34 s, 
which would indicate a performance level that does not meet MOPS criteria. 


Performance at the ABP is slightly better (in mean) than at the PTP (but with more 
spread). The standard deviation for ABP delivery error is 8.46 s, corresponding to 
performance that does not meet MOPS criteria. 


There were two types of CROSS operations performed: those with no upstream merge at 
NALTE (ABP at ZAVYO), and those with the upstream merge (ABP at NALTE). The 
latter have a maintain segment between the ABP and PTP, which would serve to deliver 
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better performance at the PTP. In fact, the PTP delivery of CROSS operations with ABP 
at NALTE has a standard deviation of 2.2 s, well within the MOPS performance. The 
PTP delivery of CROSS operations with merge near ZAVYO (ABP at ZAVYO, with 
SUBDY/UPBOB merge) has a standard deviation of 6.2 s outside of the MOPS standard. 
Some CROSS operations were performed with the ABP at ZAVYO but with the 
upstream merge at NALTE (thus no aircraft on UPBOB). Those operations have a PTP 
delivery standard deviation of 8.68 s, again outside of the MOPS standard. 


In general, for those operations that had a maintain segment (ABP at NALTE), the 
maintain performance is similar to those of MAINTAIN operations and well within the 
MOPS targets for both PTP delivery and maintain segment performance. 


CROSS operations with no maintain segment (ZAVYO ABP and PTP) are displayed in 
Figure 20. As expected, the RMS value for Maintain segment error is zero and the 
delivery statistics at the ABP and PTP are the same. However, the delivery error is worse 
than for the overall CROSS operations, indicating that the NALTE operations tended to 
perform better, owing to their large maintain segment between NALTE and ZAVYO. 
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Time Based Metrics (s) 


All Time-Based CROSS Clearances (ZAVYO): 41 
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Error (s) (nm) PTP error(s) ABPerror(s) numSpeeds Min (s) (s) 
Mean -15.69 78.56 6.20 6.20 8.07 0.45 8.14 4.99 
95% Cl low -27.58 74.19 3.60 3.60 7.18 0.40 7.17 3.04 
95% Cl high -3.80 82.93 8.80 8.80 8.97 0.50 9.11 6.93 
Range 180.00 59.29 34.22 34.22 12.00 0.80 16.20 38.14 
IQR 41.53 15.98 11.78 11.78 3.00 0.19 2.88 2.58 


Figure 20: IM Performance - Time-Based CROSS (ZAVYO) 


In fact, this is revealed looking at all CROSS operations with the ABP at NALTE as 
shown in Figure 21. The ABP delivery is worse than the PTP delivery, but overall the 
PTP delivery is better than for ZAVYO. The difference can be attributed to the late 
merging of the ZAVYO geometry, whereas all NALTE operations have a long maintain 
segment between NALTE and ZAVYO. In fact, the group of CROSS operations with the 
worst performance are those with the late merge just prior to ZAVYO when UPBOB and 
SUBDY merge as described earlier. This is attributable to several factors, including the 
lack of a maintain segment, route design and specifically the nominal vertical and speed 
profile, but also including the RF turn on the SUBDY route which has been shown to 
consistently have larger along-track wind forecast errors caused by the large change in 
track direction at altitudes corresponding to relatively unchanging forecast wind speed 
and direction. Lastly, the implementation of the TBO algorithm is known to have 
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Time Based Metrics (s) 


difficulties near the PTP in the time interval where the lead has crossed the ABP and the 
FIM aircraft is entering the segment of the trajectory that requires a nominal deceleration 
to the final stable approach speed. 


All Time-Based CROSS Clearances (NALTE): 27 
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Figure 21: IM Performance - Time-Based CROSS (NALTE) 


Time-Based SPACE Operations 


Lastly, we report statistics for time-based SPACE operations (Figure 22). There were 
only five time-based SPACE operations and thus the statistics are reported in tabular 
form only. All SPACE operations were performed with either the 757 or 737 as lead (no 
F900) on either merging or non-merging geometries with the lead on either in case of 
merges. Because of the very small number of experiments attempted, the statistics are not 
broken down by geometry. SPACE operations are special-case CROSS operations where 
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the ABP and PTP are the same, and computed by the system at a distance of 6.25 nm 
from the runway threshold. No procedure is entered in the system; the geometry is 
assumed to be along the runway’s extended centerline. When a FIM aircraft merges onto 
final approach, the merge point is part of the geometry and is computed by the system. 
Delivery statistics for SPACE operations are similar to those of CROSS, with slightly 
less speeds issued. Although within the MOPS performance targets, the sample size is 
small and thus the confidence interval for the delivery performance metric is larger than 
other clearance types. 


InitialSpacing IM Length numSpeeds_per pilotTime_mean pilotTime_std 
Error (s) (nm) PTP error(s) ABPerror(s) numSpeeds Min (s) (s) 
Mean -10.62 21.16 3.37 3.37 5.00 0.84 7.20 3.14 
95% Cl low -38.44 12.62 -1.33 -1.33 3.76 0.42 -0.87 -2.88 
95% Cl high 17.19 29.69 8.08 8.08 6.24 1.27 15.27 9.16 
Range 52.46 7 o7) 9.90 9.90 2.00 0.88 17.00 11.75 
IQR 17.77 8.07 5.15 5.15 2.00 0.28 8.75 3.35 


Figure 22: IM Performance - Time-Based SPACE (5 samples) 


Distance-Based Operations by Clearance Type 


Finally, we inspect the statistics for distance-based operations by clearance type. Because 
of the smaller sample size of those operations, the statistics are delivered in table form 
only (Figure 23 through Figure 26). No true comparison of delivery across distance and 
time-based operations can be performed without normalization caused by the difference 
in units. No normalization was attempted for this study. The majority of the distance - 
based operations performed were done en route, with the exception of the CROSS and 
SPACE clearances. 


InitialSpacing IM Length PTP error numSpeeds__pilotTime_mean MaintainError 
Error (nm) (nm) (nm) numSpeeds perMin (s) pilotTime_std (s) _RMSS (s) 
Mean 0.07 30.11 -0.31 2.67 0.90 6.78 0.96 0.14 
95% Cl low -0.09 4.59 -1.01 -1.13 -0.76 0.45 -1.23 0.10 
95% Cl high 0.23 55.63 0.40 6.46 2.57 13.10 3.15 0.18 
Range 0.12 18.84 0.50 3.00 1.34 5.00 1.73 0.03 
IQR 0.09 14.13 0.38 2.25 1.00 3.75 1.30 0.03 


Figure 23: IM Performance - Distance-Based MAINTAIN (Three Samples, all A Scenarios) 


InitialSpacing IM Length PTP error numSpeeds__ pilotTime_mean MaintainError 
Error (nm) (nm) (nm) numSpeeds perMin (s) pilotTime_std (s) _RMSS (s) tCapture (s) 
Mean -0.62 31.09 0.52 3.00 0.73 10.37 3.70 9.52 285.00 
95% Cl low -41.17 -24.20 -12.96 -9.71 -0.14 -38.86 -34.38 
95% Cl high 39.94 86.38 13.99 15.71 1.61 59.61 41.79 
Range 6.38 8.70 FLAW) 2.00 0.14 7.75 5.99 0.00 0.00 
IQR 6.38 8.70 2.12 2.00 0.14 7.75 5.99 0.00 0.00 


Figure 24: IM Performance - Distance-Based CAPTURE (Two Samples, all A Scenarios) 
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InitialSpacing IM Length 


Error (nm) (nm) 
Mean 0.91 72.75 
95% Cl low -1.62 54.51 
95% Cl high 3.45 90.99 
Range 6.95 59.52 
IQR 4.73 20.28 


PTP error 
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Figure 25: IM Performance - Distance-Based CROSS (Seven Samples, all B Scenarios) 


InitialSpacing IM Length PTP error ABP error numSpeeds_per pilotTime_mean_pilotTime_std 
Error (nm) (nm) (nm) (nm) numSpeeds Min (s) (s) 
Mean -0.84 10.06 0.09 0.09 3.00 1.08 5.33 2.30 
95% Cl low -2.61 5.96 -0.49 -0.49 0.52 -0.17 -3.39 0.39 
95% Cl high 0.94 14.16 0.68 0.68 5.48 2.33 14.06 4.21 
Range 1.43 3.06 0.45 0.45 2.00 0.98 7.00 1.41 
IQR 1.07 2.30 0.34 0.34 1.50 0.74 5.25 1.06 


Figure 26: IM Performance - Distance-Based SPACE (Five Samples) 


Along-Track Wind Forecast Error 


Along-Track Wind Forecast Error (ATWFE) is computed for all flights for both ownship 
and traffic and the RMS values are computed individually and for their difference (traffic 
—ownship). Figure 27 reports the ATWFE metrics for each flight day. Blue squares are 
ATWEE computations for ownship, red triangles for traffic, and green squares for the 
difference between the two. In cases where ownship and traffic are on the same path, the 
differences in ATWE should be minor as the wind experienced by both aircraft may only 
be slightly different because of the actual flown altitude variations and a minor time 
difference between both aircraft being at equivalent 2-D positions. Thus, some green 
squares and red triangles are nearly superimposed in the figure. 


Clearly, ATWFE will vary by geometry and time of flight for a given experiment because 
the experienced wind changes. As the figure shows, there are even large variations in the 
ATWEE within a given day (e.g., Day 7 and Day 12 have large variability). The 
descriptive statistics for each of the three ATWFE metrics is shown in the table in Figure 
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Flight Day 
ATWFE 
Ownship ATWFE ATWFE Diff 
(kts) Traffic (kts) (kts) 
Mean 8.21 7.86 9.14 
95% Cl low 7.56 7.27 8.50 
95% Cl high 8.87 8.46 9.79 
Range 21.64 17.72 21.60 
1QR 4.47 4.43 4.91 


Figure 27: Along-Track Wind Forecast Error 


The experienced wind can have an effect on the IM Operation (headwind or tailwind). 
The along-track component of the actual wind is computed for all flights for both 
ownship and traffic and the mean of the experienced along track wind component is 
computed and shown for each day in Figure 28. Again, there is large variability within 
each day caused by the many different geometries flown and the four-hour time span of 


most daily campaigns. 
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Ownship Along Track True wind mean [kts, pos=tailwind] 


150 


100 


50 


-50 


-100 


0 2 4 6 8 10 12 14 16 18 


REV A D780-10416-1 Page 43 of 116 


Flight Day 


True wind True wind 
alongtrack along track 
Mean (kts) - Mean (kts) - 


Ownship Traffic 
Mean -10.83 -9.69 
95% Cl low -14.95 -13.65 
95% Cl high -6.71 -5.74 
Range 197.87 194.69 
IQR 27.44 23.34 


Figure 28: Along Track True Wind (ownship) 


Along-track wind forecast error does not appear to be strongly correlated to the mean of 
the along track experienced wind. To enable this comparison the mean of the experienced 
wind must be rendered positive (RMS values for along track wind forecast error are 
always positive) first by taking its absolute value. The correlation coefficient between 
ATWFE and along-track true wind (abs) is 0.27. 


20 


Fuel Burn 


Average fuel burn rate statistics per clearance type is shown in Figure 29. Rates are 
comparable across operation type, except for the SPACE clearance which has a higher 
burn rate owing to its low-altitude, mostly level operational regime. Differences by other 
grouping variables are treated in the ANOVA section (Section 5). 
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Figure 29: Average Fuel Burn Rate by Operation 


Flight Technical Error 


Flight Technical Error (FTE) is defined as the deviation of the aircraft (laterally and 
vertically) from its guidance path, embodied by a generated trajectory. The FMS and the 
FIM system both derive a generated trajectory for the basis of their control functions. 
Thus, four FTE metrics are computed: 

e FMS Lateral deviation 

e FMS Vertical deviation 


e FIM Lateral deviation 
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e FIM Vertical deviation 


As noted in Annex C, the actual generated trajectory changes throughout the flight and 
thus the FTE metrics represent the instantaneous deviation from the current generated 
trajectory. There can be some discontinuity in the FTE deviation data at times of 
trajectory generations. The FMS does not export its trajectory, only the instantaneous 
deviations. The FIM system does log its trajectory and the instantaneous deviations. 


Lateral FTE is a measure of how close to the generated lateral path the aircraft actually 
flew. Given that none of the FIM operations have control in the lateral dimension, and 
that all flights in the flight test were performed in LNAV or ILS approach guidance 
(localizer), the lateral FTE metrics are not expected to be large or vary largely. 


Vertical FTE is a measure of how close to the generated vertical path the aircraft actually 
flew. In FIM descent operations under the ATD-1 concept and associated FIM prototype, 
speed control is achieved mostly in the VNAV mode, in speed on elevator mode. Thus 
vertical deviations are expected in regions where speeds must be altered significantly 
from the assumed nominal speed profile. Pilots were instructed to try to remain close to 
the original vertical path and the vertical FTE metric is used to gauge that effort. 


As noted in section | only the 737 recorded data for the FMS lateral and vertical 
deviations. Thus, the FMS FTE for the 757 contains no data. Figure 30 shows the 
statistics for FMS lateral FTE by path flown. The final approach path has larger means 
for FMS lateral FTE than the other paths. That is because the path is defined as a straight 
line along the extended runway centerline but some of the final approach spacing 
geometries had the aircraft approach from the right at a 30-degree intercept angle. Thus 
in the period prior to the intercept the aircraft was flying on heading or track, and the 
lateral deviation to the extended centerline is initially large. The en route path has a 
comparable mean to the descent paths but with more spread. This is likely caused by slop 
in the initial conditions where, in some cases, the FIM operation was started just before 
or just after ZIRAN, where a turn occurs. The rest of the geometry is straight. 
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FMS Lateral Deviation RMS (nm) 
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Figure 30: FMS Lateral Deviation RMS by Path 


By comparison, the FIM Lateral Deviation RMS by Path is shown in Figure 31. This data 
includes the 757. The en route path has larger deviations resulting from a flight test 
procedure. Crews were instructed to not enter SINGG in the flight plan in the FMS to 
protect the Military Operations Area (MOA) to the NW of the path. However, SINGG 
was the PTP for this path and thus the lateral FTE of the FIM system (which includes 
SINGG) grows larger as the aircraft approaches SINGG. The final approach path does 
not have significantly larger deviations than the other paths because the FIM system 
draws the complete path from FIM initiation to the runway (including an intercept if 
required) even when the aircraft may be flown in heading or track mode against the FMS 
path which only includes the runway extended centerline. 
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FIM Lateral Deviation RMS (nm) 
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Figure 31: FIM Lateral Deviation RMS by Path 


Vertical deviation FTE for the FMS is summarized in Figure 32. The en route path 
appears to have a consistent 4000 ft difference from the FMS nominal altitude. Given that 
this path is flown at level cruise altitude of 35,000 ft, the difference likely is caused by 
the FMS vertical altitude reference being the ECON altitude, not the actually flown 
altitude driven by the MCP. This is expected if the crews did not, in fact, enter the cruise 
altitude in the FMS VNAV path, but left it as an ECON cruise and flying the aircraft 
vertically using the MCP. Because no detailed FMS data is available for review, this 
theory cannot be confirmed. The average vertical deviation throughout the flight (RMS 
value) is similar for all three descent paths. This deviation is mostly caused by the aircraft 
deviating from the FMS guidance to execute FIM speeds using the MCP as detailed 
earlier. The fact that the deviation is generally not much larger than 2000 ft is promising 
and a product of the containment of the FIM speed control authority to within 15% of the 
nominal speed. The approach path has greater average deviations but again, this is caused 
by an artifact of the procedure where the vertical reference for the FMS is along the 
approach path with a glideslope by the portion of the IM operation outside of the runway 
centerline was conducted mostly level. 
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FMS Vertical Deviation RMS (ft) 
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Figure 32: FMS Vertical Deviation RMS by Path 


Vertical deviation FTE for the FIM system is summarized in Figure 33. In this case, the 
FIM system generates cruise and approach vertical paths based on IM operational inputs 
in the FIM system and corresponds more closely to the flown operation than the FMS 
path. The FIM system generates a vertical trajectory assuming a kinematic aircraft model 
whereas the FMS system has full aerodynamic and engine performance at its disposal. 
The fact that the FIM system generates vertical trajectories closer to what was flown is an 
artifact of the procedure design and the difference in fidelity between the FMS and FIM 
vertical trajectory construction models. 
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Figure 33: FIM Vertical Deviation RMS by Path 


Path Definition Error 


Path Definition Error (PDE) is defined as the difference between the FMS-generated 
trajectory and the FIM system-generated trajectory in the lateral and vertical dimensions. 
The FMS trajectory is not directly available for comparison, thus the PDE metric is based 
on the difference between the FMS-reported deviations and the FIM-reported deviations 
(lateral and vertical) as explained in Annex C of the Flight Test Plan. Again, because the 
757 did not record data for FMS deviations, the PDE metric cannot be reported. 


The summary statistics for lateral PDE by path are shown in Figure 34. The larger lateral 
PDE for the en route path is again caused by the difference in path entry between the 
FMS and FIM system, a procedure the pilots followed to avoid getting close the NW 
MOA near SINGG and the downstream turn through JELVO. All other paths have 
average lateral PDEs of very small value (less than an aircraft wingspan) with relatively 
small dispersion. 


The summary statistics for vertical PDE by path are shown in Figure 35. Again, the en 
route path highlights a difference between the FMS programmed cruise altitude and the 
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entered FIM system altitude. This difference did not have a bearing on the operation 
because the aircraft flew MCP altitude level at the FIM entered altitude. As discussed 
earlier, the final approach PDE is mostly caused by the FMS vertical profile generated 
along the runway extended centerline in operations that may have an intercept section or 
even a level flight segment at IM onset until approach path capture. The descent paths 
have average vertical PDE of less than 2000 ft, which is a relatively small number given 
the difference in fidelity between the FMS and FIM system vertical trajectory generators. 
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Figure 34: Lateral PDE RMS by Path 
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Figure 35: Vertical PDE RMS by Path 
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5. Analysis of Variance 


The purpose of conducting ANOVA on the dataset is to identify statistically meaningful 
differences between groups in the data. It is important to realize that the flight test matrix 
in no way represents a full-factorial design and thus not all effects will possibly be 
discernable. This means that many of the interaction effects between the factors may not 
be captured. Thus, one-way ANOVA is considered exclusively in this report. Multiway 
ANOVA is not performed here, but linear regression models are attempted and reported 
in section 6; derivation of such models utilizes the same statistical techniques as 
performed using ANOVA. 


Furthermore, because much of the data are not normally distributed, nonparametric 
means of performing ANOVA are sought, such as Kruskal-Wallis tests for comparing 
samples of unknown mean and variance and testing whether they may belong to the same 
distribution. All statistical tests are conducted at a significance level of 95% (alpha level 
of 0.05) and statistical significance is reported at that level. 


Are there differences in ATWFE for ownship, traffic and their difference, 
between days and geometry? 


ATWEE ownship 

By day: (p value = 3.86E-9) Significant differences at 95% confidence level 
By path: (p value= 0.0927) No significant difference at 95% confidence level 
ATWEE traffic 

By day (p value = 8.61E-12) Significant differences at 95% confidence level 
By path (p value= 0.0007) Significant differences at 95% confidence level 
ATWEHE difference 


By day (p value = 1.1e-5) Significant differences at 95% confidence level 


Days 7, 14, and 16 have significant differences in ATWFE from all other days as shown 
in Figure 36. 
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Figure 36: ATWFE By Flight Day 


The final approach path (used on SPACE clearances) has significantly different ATWFE 
than the other paths (Figure 37). This is likely caused by the low altitude nature of this 
path and its much straighter nature. This path does not include the RF turns present in the 
transition to the final approach present in the other paths, and thus is subject to less 
forecast error observed for all other paths (see section 2) through those turns. 
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Figure 37: ATWFE By Ownship Path 
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Average Fuel Burn Rate (Ibs/hr) 


Are there differences in Average Fuel Burn rate between operations, 
geometry, and aircraft type? 


By operation units: (p value= 0.0067) Significant differences at 95% level 


Distance-based operations have higher average fuel burn rate (Figure 38) likely caused by 
the higher IM Speed rate as will be seen later in this section. 
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Figure 38: Average Fuel Burn Rate by IM Operation Units (time/distance) 


By clearance type: (pvalue=0.0012) Significant differences at 95% level 
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The main statistically significant difference is between CROSS and SPACING operations 
(Figure 39). SPACING operations are operated at low altitudes on generally level 
segments and thus are expected to burn more fuel. 
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Figure 39: Average Fuel Burn Rate by Clearance Type 
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In general the 757 burns more fuel, as is expected because of the different weight class 
and age (generation) of aircraft (Figure 40 and Figure 41). 


By aircraft type: (p value = 3.54E-06) 
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Figure 40: Average Fuel Burn Rate by Ownship Aircraft Type 
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By path: (p value = 1.86E-08) Significant differences at 95% level 
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Figure 41: Average Fuel Burn Rate by Ownship Path 


The en route and final approach paths have higher fuel burn rate, which is expected given 
that they are mostly level flight operations and thus will typically operate with speed on 
throttles. The other paths are used in descent operations in VNAV mode with speed on 
pitch, a more efficient way to modify speed. The UPBOB path appears to only have 
statistically significantly less burn than the JELVO path, because of the design of its 
nominal vertical and speed profile and generally shorter length during IM operations 
(Figure 42 and Figure 43). 
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A different view of the same effect is seen when looking at difference by operation types: 


By operation type: (p value= 1.46E-07) — Significant differences at 95% level 
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Figure 42: Average Fuel Burn Rate by Operation Type (A/B/C) 
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Average Fuel Burn Rate (lbs/hr) 


By chain position: (p value = 1.424E-7) Significant differences at 95% level 
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Figure 43: Average Fuel Burn Rate by Chain Position 


There is significant difference in average fuel burn by chain position. Given that there is 
no statistically significant differences in IM Speed rate by chain position (see later 
section), this difference results from the fact that most second chain positions were flown 
by the 737, which has already been shown to burn less fuel. 


Are there differences in FMS lateral FTE by day, path, or chain position? 


By day: (p value =0.5552) No significant differences at 95% level 
By path: (p value= 0.0025) Significant differences at 95% level 


By chain position: — (p value = 0.075) No significant differences at 95% level 
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As discussed in section 4 there are differences in FMS lateral FTE across paths and the 
ANOVA reveals them to be statistically significant. However, the only pairwise 
statistical differences are between the en route and final approach paths compared to the 
descent paths, which were explained earlier. No statistically meaningful differences exist 
between the descent paths. 


The lack of statistically meaningful difference by day is a testament to the performance of 
LNAV lateral control in presence of a wide range of wind conditions. 


Are there differences in FMS vertical FTE by day, path, or chain position? 


By day: (p value = 0.6457) No significant differences at 95% level 
By path: (p value= 0.0022) Significant differences at 95% level 


By chain position:  (p value=0.2231) No significant differences at 95% level 


As discussed in section 4 there are differences in FMS vertical FTE across paths and the 
ANOVA reveals them to be statistically significant. However, the only pairwise 
statistical differences are between the en route and final approach paths compared to the 
descent paths, which were explained earlier. No statistically meaningful differences exist 
between the descent paths. 


Are there differences in FIM lateral FTE by day, path, or chain position? 


By day: (p value = 0.383) No significant differences at 95% level 
By path: (p value = 1.03E-11) Significant differences at 95% level 
By aircraft: (p value= 0.0016) Significant differences at 95% level 


By chain position: —(p value = 0.002) Significant differences at 95% level 


As discussed in section 4 the difference in FIM lateral FTE across path is driven by the 
large differences in the cruise and approach paths that were explained. When comparing 
the FIM lateral FTE for only the B paths, the results indicate differences across the 
descent paths. 


By path (B runs only):(p value = 5.2E-7) Significant differences at 95% level 
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Figure 44: Lateral FTE RMS (FIM) by Descent Paths 


The only pairwise meaningful difference is between UPBOB1 and the two other SUBDY 
paths. This difference is attributed to the relatively lower lateral complexity of the 
UPBOB path, especially because IM operations were started after the first large turn and 
the remainder of the path is generally straighter than either of the SUBDY paths, which 
have a general change of direction of more than 90 degrees in its upper portion, and RF 
turns in its latter portions. Possibly the statistics reveal differences between how the FIM 
system computes turns compared to the FMS because there were no observable 
differences in lateral FTE between the descent paths for the FMS (Figure 44) and the 
ANOVA reveals them to be statistically significant. However the only pairwise statistical 
differences are between the en route and final approach paths compared to the descent 
paths, which were explained earlier. No statistically meaningful differences exist between 
the descent paths. 


There are differences in FIM Lateral FTE by aircraft but the difference is so small it is 
likely not contributing to any FIM performance (Figure 45). 
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Figure 45: FIM Lateral FTE by Aircraft 


Are there differences in FIM vertical FTE by day, path, or chain position? 


By day: (p value= 0.0005) Significant differences at 95% level 
By path: (p value = 2.82E-6) Significant differences at 95% level 
By aircraft: (p value = 0.014) Significant differences at 95% level 


By chain position:  (p value=0.0522) No significant differences at 95% level 


There are differences in FIM Vertical FTE by day, which is expected given that the flight 
path angles flown by aircraft when in VNAV speed mode vary based on the experienced 
wind for the same CAS value (Figure 46). 
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The differences by path are dominated by the differences between the en route path and 
the approach paths that are described before. When comparing only among the descent 
(B) paths, the test still reveals statistical differences: 


By path (B only): (p value = 0075) Significant differences at 95% level 
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Figure 46: FIM Vertical RMS by Descent Paths 


The JELVO and ZIRAN paths have differences among each other but not from UPBOB. 
Both SUBDY transitions have a common segment downstream of NALTE and thus the 
difference is expected to come from the portion upstream of NALTE. However the 
difference is small (less than 200 ft) and is not worth investigating further (Figure 47). 
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There are differences in FIM Vertical FTE by aircraft but the difference is so small it is 
likely not contributing to any FIM performance. 
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Figure 47: FIM Vertical FTE RMS by Aircraft 


Are there differences in Lateral PDE by day, path, or chain position? 


By day: (p value = 0.5362) No significant differences at 95% level 
By path: (p value = 2.82E-6) Significant differences at 95% level 


By chain position: (p value=0.8367) No significant differences at 95% level 
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Again the path differences are dominated by the difference between the en route and 
approach paths (Figure 48). The difference across descent paths is statistically significant 
(p value = 0.0252) and shows more variability in PDE across the ZIRAN transition than 
the other paths. 
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Figure 48: Lateral PDE by Descent Path 


Are there differences in Vertical PDE by day, path, or chain position? 


By day: (p value= 0.4071) No significant differences at 95% level 
By path: (p value = 0..002) Significant differences at 95% level 


By chain position: (p value= 0.3487) No significant differences at 95% level 


Once again the difference by path is dominated by the difference between the en route 
and approach paths (Figure 48) There is no statistically significant difference across the 
descent paths (p value = 0.1027). 
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Are there differences in time-based PTP delivery across clearance types, 
geometries, and aircraft type? 


PTP delivery (time) 


By clearance type: (p value= 0.0022) Significant differences at 95% level 
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Figure 49: Time-Based PTP Error By Clearance Type 


CROSS clearances have significantly worse performance at the PTP than MAINTAIN, 
and no other pairwise comparisons reveal statistically significant differences. It appears 
SPACING clearances have worse performance (Figure 49), but the low number of 
samples render this a visual artifact. The notches on the boxplot can be used to inspect 
pairwise differences. Overlap of notches indicates statistically meaningful pairwise 
difference. In the figure, onlt the notches of CROSS and MAINTAIN do not overlap. 
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By ownship path: 


(p value = 3.43E-07) 
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Figure 50: Time-Based PTP Error by Ownship Path 
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The UPBOB path flown by ownship has significantly worse performance than other 
geometries (Figure 50). It should be noted that UPBOB was never flown in-trail with 
another aircraft on UPBOB, thus all UPBOB performance is attributable to the exercise 
of a CROSS clearance, which has already been determined to have worse performance. 
UPBOB did not show significant difference in ATWFE (Figure 37) thus, the effect of the 
UPBOB path is not rooted in the wind forecast error, but likely in the design of the 
procedure itself (nominal vertical or speed profile), or its use predominantly in CROSS 
operations with a late merge near the PTP. 
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By traffic path: (p value = 0.0436) Significant differences at 95% level 


20 


oa 


PTP error(s) 


5 
ZIRAN.BARYN.SINGG.JELVO ZIRAN.SUBDY1 JELVO.SUBDY1 final_appr 


Figure 51: Time-Based PTP Error by Traffic Path 


The delivery performance of pairs involving a lead on the en route path is significantly 
different. It should be noted that this should only happen when ownship is en route as 
well (no CROSS clearances flown en route) and thus the difference by lead path is 100% 
attributable to the difference by operation type as shown in Figure 51. 
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By scen. type: (A/B/C) (p value =0.0189) Significant differences at 95% level 


With A runs showing different performance than B or C runs (no difference between B 
and C runs). 
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Figure 52: Time-Based PTP Error By Scenario Type 


By ownship aircraft: (p value= 0.0523) No significant differences at 95% level 


By traffic aircraft: (p value =0.0521) No significant differences at 95% level 


By chain position:  (p value=0.1925) No significant differences at 95% level 


Given the small sample size of distance-based clearances, there are no statistically 
meaningful differences between distance-based PTP performance across clearance type, 
geometry, or aircraft type (Figure 52). 
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Are there differences in the IM speed rate across clearance types, 
geometries and aircraft type? 


IM speed rate 
By clearance type: (p value= 0.0016) Significant differences at 95% level 


MAINTAIN operations have significantly more IM speeds issued per minute of IM 
operation than other clearance types (with the exception of SPACING, with a very small 
sample). This is due to a condition when the spacing error causes a speed error 
discretization oscillation, as is often the case when the non discretized speed error is on 
the boundary of the discretization value (e.g. 10kts). Hysterisis requirements exist in the 
system and its implementation is being reviewed in light of the observed behavior. 


SPACING operations also have a higher average IM Speed Rate, although not 
statistically significant because of the low sample size and large spread (Figure 53). This 
is likely caused by the shorter nature of the operation, which tends to generate higher 
rates for mathematical reasons. 
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Figure 53: IM Speed Rate by Clearance Type 
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By chain position:  (p value= 0.8533) No significant differences at 95% level 


By units: (p value = 0.045) Significant differences at 95% level 
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Figure 54: IM Speed Rate by Operation Unit Type (TIME versus DISTANCE) 


Distance-based operations appear to have more IM speeds issued per minute than time- 
based operations (Figure 54). This is caused by the lack of decceleration estimation in the 
CTD implementation. In distance operations the CTD algorithm follows the traffic “on 
the string” and responds to its speed changes directly. 
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IM Speed Rate (per min) 


By ownship path: (p value=0.0002) Significant differences at 95% level 
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Figure 55: IM Speed Rate by Ownship Path 


The UPBOB path has significantly less IM speed rate than other paths (Figure 55). This 
is attributable to the fact that it is used only in CROSS operations, while the SUBDY path 
is also used with MAINTAIN and CAPTURE operations, which tend to have higher IM 
speed rates as shown in Figure 55. 
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IM Speed Rate (per min) 


By traffic path: (p value=0.0178) Significant differences at 95% level 
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Figure 56: IM Speed Rate by Traffic Path 


The only statistically significant pairwise difference involves the final approach path, 
which has a higher IM Speed Rate (Figure 56). This path is used only in SPACING 
operations, which has already been shown to have higher IM Speed Rates than other 
operations. 


By aircraft type: (p value=0.503) No significant differences at 95% level 


By lead aircraft type: (p value=0.258) No significant differences at 95% level 
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Are there differences in pilot implementation times (mean and std) between 
aircraft? 


Pilot time mean (all): By aircraft (p value = 3.1le-5) Significant differences at 95% 
confidence level 


Pilot time std (all): — By aircraft (p value = 0.295) No significant differences 
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Figure 57: Pilot Implementation Time by Aircraft 


The 757 took significantly more time than the 737 to implement IM clearances (on 
average 3 seconds longer) (Figure 57). This difference is difficult to attribute because it is 
rooted in individual pilot technique, which varies by crew, by day, and by affiliation. 
Note that the comparison of MCP time dispersion is done on the means of the MCP time 
standard deviations, computed for each run. Thus it is not direct comparison of variance, 
rather a comparison of means of a metric, that indicates the spread of speed 
implementation times across an individual run. 
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Among time-based CROSS operations, are there differences in 
performance by geometry, aircraft type, or chain position? 

PTP delivery 

By ownship path: (p value= 2.9E-05) Significant differences exist at 95% level 
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Figure 58: Time-Based CROSS, PTP Error by Ownship Path 


Once again UPBOB displays worse performance (Figure 58). This was already seen and 
described in the overall PTP performance and caused by the UPBOB geometry design 
and the fact that it is always executed as pair number 2 in a late merge (at ZAVYO) 
situation. 
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By ABP placement: (p value = 4.91E-06) Significant differences at 95% level 
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Figure 59: Time-Based CROSS, PTP Error By ABP Placement 


At the PTP, performance is worse when the ABP is at ZAVYO and thus a late merge 
with no maintain segment (Figure 59). The maintain segment when the ABP is placed at 
NALTE helps reduce the variability and increase the delivery performance at the PTP. 


By traffic path: (p value = 0.6908) No significant differences at 95% level 


By ownship aircraft: (p value = 0.8583) No significant differences at 95% level 
By lead aircraft: (p value = 0.4688) No significant differences at 95% level 


By chain position: —(p value = 0.63) No significant differences at 95% level 


Some time-based CROSS operations were performed with the ABP at ZAVYO with two 
aircraft on SUBDY. Those operations had no maintain segment and performed 
statistically worse at the PTP than the same geometry with the ABP at NALTE. In 
general, again the performance at the PTP in cases where the ABP is at ZAVYO is worse 
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for those pairs involving UPBOB, which also happens to always be in third position 
(second pair). 


ABP delivery 
By ownship path: (p value= 0.0001) Significant differences exist at 95% level 
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Figure 60: Time-Based CROSS, ABP Delivery by Ownship Path 


Once again significant differences with UPBOB are displayed, which was only used in 
situations with ABP and PTP collocated at ZAVYO and thus reflects the same effect as 
that of PTP delivery performance (Figure 60). 
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By ABP placement:(p value = 4.54E-05) Significant differences at 95% level 
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Figure 61: Time-Based CROSS, ABP error By ABP Placement 


Performance at the ABP is worse in cases with a late merge (ZAVYO), likely also 
compounded by the poorer performance of the UPBOB path (Figure 61). 


By traffic path: (p value = 0.8876) No significant differences at 95% level 


By ownship aircraft: (p value = 0.7224) No significant differences at 95% level 


By lead aircraft: (p value = 0.6152) No significant differences at 95% level 


By chain position: (p value = 0.938) No significant differences at 95% level 


IM Speed Rate 


By ABP placement: (p value = 7.88E-06) Significant differences at 95% level 
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Figure 62: Time-Based CROSS, IM Speed Rate by ABP placement 


CROSS operations with ABP placement at NALTE result in a higher speed rate, which is 
attributable to the maintain segment because the MAINTAIN operations have already 
been shown to have higher IM speed rates than other clearances (Figure 62). 
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By ownship path: (p value = 5.44E-05) Significant differences at 95% level 
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Figure 63: Time-Based CROSS, IM Speed Rate by Ownship Path 


The ZIRAN transition of SUBDY seems to have consistently issued more speeds per 
minute than other paths (Figure 63). This is caused by the fact that 87.5% of the cases 
where ownship is on the ZIRAN path occurs for an operation with the ABP at NALTE 
and thus has a maintain segment, which will generate more IM speeds, as has been shown 
earlier. 


By traffic path: (p value= 0.3759) No significant differences at 95% level 


By ownship aircraft: (p value= 0.4271) No significant differences at 95% level 


By lead aircraft: (p value =0.2591) No significant differences at 95% level 


By chain position: —(p value = 0.63) No significant differences at 95% level 
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Among time-based CAPTURE operations, are there differences in 
performance by geometry, aircraft type, or chain position? 


PTP delivery 
By path: (p value=0.1219) No significant differences at 95% level 
By aircraft type: (p value=0.0176) Significant differences at 95% level 
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Figure 64: Time-Based CAPTURE, PTP Error by Aircraft Type 


It is difficult to explain a difference in PTP delivery by aircraft type for the CAPTURE 
operation when no difference was observed by aircraft type among all time-based 
operations (Figure 64). No CAPTURE operations were performed on the UPBOB path 
and therefore the difference cannot be rooted in route design. There is no difference in the 
maintain segment performance between aircraft type, nor is there an observed difference 
in the time to capture. However, it is believed that in general the 757 has different 
acceleration and deceleration performance than the 737 resulting in this performance 
difference. In fact, later analysis shows that the capture rate of the 757 is higher than the 
73d: 
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By lead aircraft: (p value = 0.0928) No significant differences at 95% level 


By condition (A/B): (p value= 0.0404) Significant differences at 95% level 
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Figure 65: Time-Based CAPTURE, PTP Error by Scenario Type 


Cruise operations are shorter in time and have less control authority than descent 
operations, resulting in operations that tend to arrive earlier (Figure 65). 


By chain position: = (p value = 0.133) No significant differences at 95% level 


tCapture 


The time to capture does not significantly differ across all variations of execution of the 
time-based CAPTURE clearance throughout the flight test. 


By path: (p value = 0.5753) No significant differences at 95% level 


By ownship aircraft: (p value= 0.5558) No significant differences at 95% level 


By lead aircraft: (p value = 0.3456) No significant differences at 95% level 


By chain position: (p value=0.4773) No significant differences at 95% level 
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By condition (A/B): (p value= 0.3054) No significant differences at 95% level 


Capture Rate 


However, when looking at capture rate instead of the time to capture, there are significant 
differences among some of the groups. See section 4 for a defintion of how capture rate is 
computed. However, indendepent effects are difficult to attribute because more than half 
the observations for time-based CAPTURE operations are for the 757 on JELVO against 
the F900, thus in chain position 1. Each of those attributes in isolation has a statistically 
significant higher capture rate (Figure 66 through Figure 69). It should be noted that the 
attribution of aircraft to lead position is not symmetrical in the flight test and this could 
skew those results. For example, 45.8% of FIM pairs have a B757 lead, 10.42% have a 
B737 lead, and 43.8% have a F900 lead). 


By ownship aircraft: (p value= 0.0203) Significant differences at 95% level 
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Figure 66: Capture Rate by Aircraft Type 
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By path: (p value = 0.0045) Significant differences at 95% level 
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Figure 67: Capture Rate by Ownship Path 
By lead aircraft: (p value = 0.0028) Significant differences at 95% level 
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Figure 68: Capture Rate by Lead Aircraft Type 
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By chain position: —(p value = 0.002) Significant differences at 95% level 
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Figure 69: Capture Rate by Chain Position 


By condition (A/B): (p value=0.0516) No significant differences at 95% level 


maintain RMS 


By path: (p value= 0.1063) No significant differences at 95% level 


By ownship aircraft: (p value=0.2175) No significant differences at 95% level 


By lead aircraft: (p value =0.1327) No significant differences at 95% level 


By chain position:  (p value=0.0314) Significant differences at 95% level 
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maintain RMS (s) 


Figure 70: Time-Based CAPTURE, Maintain RMS by Chain Position 


Operations at the last position in the chain have worse maintain segment performance 
(Figure 70). This can be attributed to the chain effect and associated accumulation of 
delay but, interestingly, does not translate to discernable differences in PTP delivery as 
was discussed earlier. 
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By condition (A/B): (p value =0.0404) Significant differences at 95% level 


maintain RMS (s) 


Figure 71: Time-Based CAPTURE, maintain RMS by scenario type 
Cruise operations have worse maintain segment performance than descent operations 
(Figure 71). This is likely the same effect that was observed in PTP delivery 
performance. 


IM Speed Rate 


IM Speed Rate does not significantly differ across all variations of execution of the time- 
based CAPTURE clearance throughout the flight test. 


By traffic path: (p value = 0.8994) No significant differences at 95% level 


By ownship aircraft: (p value= 0.9267) No significant differences at 95% level 


By lead aircraft: (p value = 0.579) No significant differences at 95% level 


By chain position: (p value= 0.4405) No significant differences at 95% level 


By condition (A/B): (p value=0.7144) No significant differences at 95% level 
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Across time-based MAINTAIN operations, are there differences in 
performance by geometry, aircraft type, and chain position? 

PTP delivery 

By path: (p value = 0.6704) No significant differences at 95% level 


By ownship aircraft: (p value= 0.0063) Significant differences at 95% level 
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Figure 72: Time-Based MAINTAIN, PTP Error by Aircraft Type 


PTP delivery differences between aircraft types is difficult to explain, especially because 
the maintain segment performance does not exhibit statistically significant differences 
(Figure 72 and Figure 73). This is likely the product of late adverse spacing evolution 
being too small to significantly affect the entire maintain segment performance. It has 
been shown that the along-track wind forecast error is consistently large through the 
operation of the RF turns present on the SUBDY path. Given the fact that MAINTAIN 
operations have predominantly been performed on the SUBDY path and that the RF turns 
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occur late in the operation, likely near the time the lead aircraft reaches the PTP, this 
would tend to support that theory. 


By chain position: (p value= 0.0068) Significant differences at 95% level 


PTP error (s) 


Figure 73: Time-Based MAINTAIN, PTP Error by Chain Position 


By condition (A/B): (p value = 0.6704) No significant differences at 95% level 


Maintain RMS 


Maintain segment performance does not significantly differ across all variations of 
execution of the time-based MAINTAIN clearance throughout the flight test. 


By path: (p value= 0.3491) No significant differences at 95% level 


By ownship aircraft: (p value= 0.4528) No significant differences at 95% level 


By lead aircraft: (p value = 0.4528) No significant differences at 95% level 


By chain position:  (p value=0.2167) No significant differences at 95% level 


By condition (A/B): (p value=0.3491) No significant differences at 95% level 
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IM Speed Rate 


By path (p value = 0.0022) Significant differences at 95% level 
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Figure 74: Time-Based MAINTAIN, IM Speed Rate by Path 


The difference in IM speed rate by path is attributable to late operation along-track wind 
forecast errors through the RF turns of the SUBDY path (Figure 74). Also, given that the 
en route path is level and shorter and has less discretization of available IM speeds 
(operations in Mach rather than CAS), it should be expected to generate less IM speeds 
and worse performance, as has been seen. 


By ownship aircraft: (p value = 0.088) No significant differences at 95% level 


By lead aircraft: (p value = 0.088) No significant differences at 95% level 


By chain position:  (pvalue=0.1018) No significant differences at 95% level 
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By condition (A/B) (p value=0.0022) Significant differences at 95% level 
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Figure 75: Time-Based MAINTAIN, IM Speed Rate, by Scenario Type 


This difference describes the same effect described earlier by ownship path between the 
SUBDY and en route paths (Figure 75). 
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6. Generalized Regression Models 


Objective 


The objective for generating regression models out of the flight test data is to get a deeper 
understanding of how operational parameters at the onset of the operation affect IM 
performance. Understanding and describing what operational conditions contribute to 
each IM performance aspect is an important step in gaining confidence in the operation 
observed through the use of the prototype in the flight test. 


The set of regressors used in all regression analysis consists of the following: 


Clearance Type (MAINTAIN, CAPTURE, CROSS, SPACE) 
Intial Spacing Error (nm or s) 

Initial Altitude (ft) 

IMlength (nm) 

Aircraft Type [ownship] (737, 757) 

Lead aircraft type [traffic] (737, 757, F900) 

Chain Position (1 or 2) 

Ownship Path 

Traffic Path 

Ownship along track actual wind component mean 
Traffic along track actual wind component mean 


Chain lead (non-FIM aircraft) nominal speed profile deviation mean [delay 
condition] (kt) 


Several metrics were attempted for this regressor. The first being the test card condition, 
which includes three delay conditions for the non-FIM aircraft (no delay, medium delay, 
and high delay, see Flight Test Plan for further details). However, in many instances, the 
actual profile flown by the non-FIM aircraft did not match the desired condition. When 
used as a regressor, its inclusion tended to worsen the model or not be strong enough to 
warrant inclusion. 
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The second metric attempted was based on the actual flown CAS as compared to the 
nominal CAS profile. A mean value of that difference between tStart and tEnd is reported 
for each run. A first attempt to match the discrete test card delay conditions was made by 
clustering the resulting mean error into three clusters. The cluster analysis revealed three 
clusters for the error (-15kts, -5kts and 0 kts), but when the discrete cluster index was 
used in the regression no significant improvement was discernable. Thus the variable 
retained for this regressor in this analysis is the continuous error signal mean with 
continuous positive and negative values in kts. 


e Distance to go to the ABP at onset (CROSS only) (nm) 


Note that the along-track wind forecast error is not included in the list of regressors as it 
is unlikely to be known accurately as an operational input, even though it is likely to be a 
contributor to performance. The actual wind experienced is included separately from the 
forecast quality as it is likely estimable and is expected to have some impact on the 
performance separate from the forecast quality. In fact, several of the flown days were for 
high wind conditions that, even in the presence of a decent forecast, tended to exhibit 
worse performance. It is expected that the quality of models increases if the forecast 
errors are used as regressors, but this was not attempted. 


Regression models are attempted for this set of regressors for each of the performance 
metrics described in the ANOVA section: 


e Achieved PTP spacing error (nm or s) 

e Achieved ABP spacing error (nm or s) [CROSS only] 

e IM Speed Rate (number of IM speeds per minute of operation) 
e Time to capture (s) [CAPTURE only] 

e Maintain segment RMS error (s or nm) 


e Fuel Burn Average Rate (Ibs/hr) 


Methodology 


Standard regression techniques assume normality of the input and output data to the 
model. As described in section 5, the data collected is not necessarily normally 
distributed because of both the experiment design and the uncontrollable and non-normal 
nature of some of the input conditions (e.g., wind conditions). Thus, generalized 
regression techniques are used, that are less sensitive to the normality assumption. These 
techniques can deal with data of different distributions such as Poisson (for positive 
integer responses such as the number of IM speeds) and Gamma (for positive continuous 
responses such as the IM speed rate or the maintain segment RMS response). 
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In addition, rather than specifying a given model a priori, fitting the data to it, and then 
evaluating the fit (a process that can take a long time), stepwise model search techniques 
are employed. Matlab provides many different techniques for the construction of 
generalized linear regression models by stepwise effect inclusion. This report focuses on 
the methods that start with a simpler model and trend towards a complex model by 
iteratively adding and removing terms until the model no longer offers statistically 
meaningful improvement. 


The Matlab function used is stewiseglm with its default search convergence criterion 
based on Deviance, with is related to the statistical significance of the added term. Terms 
are added to the model if the p-value of the F or chi-squared statistic is smaller than 0.05, 
and terms are removed to the model if the p-value of the F or chi-squared statistic is 
greater than 0.05. Terms are added in order from linear to interaction terms and quadratic. 
The search thus favors simpler models, starting with purely linear terms, and ending with 
purely quadratic terms. In cases where such searches result in models with poor fit, a 
reverse search is attempted (start at the complex model and remove terms). 


The quality of the resulting model is evaluated based on the adjusted R squared value, 
related to the fit quality to a line, much like in standard regression, but adjusted for the 
number of terms in the model. R squared values range from 0 to 1, where | indicates a 
perfect fit. Regression models are reported in this document if their adjusted R squared 
value is greater than 0.7. Models with excessively high adjusted R squared values (>0.95) 
are inspected and not reported if they contain too few remaining error degrees of 
freedom. After all, it is always easy to fit a perfect line through two data points. 


Regression Models 


All Time-Based Operations 


The first models attempted were for all time-based operations, with the clearance type 
part of the list of included terms. As will be shown later, typically better submodels were 
found for each separate IM operation type. 


No overall models of time-based PTP performance were of sufficiently good quality to 
report. 


A model for the number of IM speeds with R=0.73 is described as 


Generalized Linear regression model: 
numSpeeds ~ [Linear formula with 10 terms in 7 predictors] 


Distribution = Poisson , Link = Log 


Estimated Coefficients: 


Estimate SE tStat pValue 
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(Intercept) -1.2077 0.49176 
hAtStart 4.3983e-05 9.2036e-06 
ownship path_JELVO.SUBDY1 2.5732 0.38245 
ownship path_ZIRAN.SUBDY1 2.69 0.40293 
ownship_path_UPBOB1 2.1487 0.41522 
ownship path _final_appr 2.5989 0.50943 
clearanceType CAPTURE =0;:35:97 0.091246 
clearanceType CROSS -0.28139 0.10281 
windAlongTrackComponents_mean -0.018866 0.0077476 
windAlongTrackComponents_traffic_mean -0.0089389 0.0038443 
chain_positionSTR_2 0.18182 0.08033 
traffic_speedProfileDev_mean_lead1 -0.0098503 0.0046306 
hAtStart:windAlongTrackComponents_ mean 9.9573e-07 2.7943e-07 
windAlongTrackComponents_mean*2 0.00021282 5.5598e-05 


116 observations, 102 error degrees of freedom 


0.014054 
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.7626e-06 
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.7164e-11 
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-0782e-05 


0.0061981 


0.01489 


0.020059 


0.023613 


0.033402 


0.00036594 


0.00012923 


Reports of regression models included herein will take this form and are explained in the 
next paragraph through an example. The actual model and associated data is delivered in 
an accompanying Matlab data file and associated scripts for reproduction and validation 


by NASA. 


To exercise the model, the Estimate column is used, which includes the value of the 
regression term for the factor in question. The other columns are used to report on the 
statistics of the inclusion of this term and are not used here. Because many of the 
regressor variables are categorical in nature, their terms are for specific values of those 
variables, with the nominal value built into the Intercept term. Thus in the above 
example, the nominal values for ownship path is the en route path 
(ZIRAN.BARYN.SINGG.JELVO), the nominal value for clearanceType is MAINTAIN, 
and the nominal value for chain position is 1. To exemplify the description of the model 
and its use, let us determine the estimated number of IM speeds for a CAPTURE 
operation, with ownship on the JELVO.SUBDY 1 transition, in second chain position, 
starting at 23,000 ft, with a chain lead speed deviation of -10 knots and along track winds 
of -15 knots (for both ownship and traffic). 


log(numSpeeds) = 
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-1.2077 (Intercept Term) + 

4.3983E-5 * 23,000 (hAtStart Term) + 
2.5732 (ownship_path Term) + 

-0.3597 (clearance type Term) + 
-0.018866 * -15 (ownship wind Term) + 
-0.0089389 * -15 (traffic wind Term) + 


0.18182 (chain position Term) + 
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-0.0098503 * -10 (lead speed error Term) + 

9.9573e-07 * 23,000 * -15 (hAtStart*wind interaction) + 
0.00021282 * (-15)2 (wind “2 Term) 

= 2.4192 


The log term comes from the specific distribution forced on the generalized regression, 
which for a response expected as a positive integer (as is numSpeeds) defaults to Poisson, 
thus using a log-based link function. Thus the number of IM speeds expected for this 
example would be exp(2.4192) = 11.24. 


Inspection of the model reveals interesting insight by way of the terms that are missing 
from the model (thus were not added as significant contributors). For example, it appears 
that the number of IM speeds is not affected by the length of the IM operation or the 
initial spacing error. It is mosly affected by the type of operation, wind condition, and 
lead delay condition. The initial altitude term likely serves as a surrogate for the type of 
operation (A, B, or C) because, by design, en route (A) conditions were run at FL350, 
while descent (B) conditions were started at FL230. 


Interestingly, no good model can be found for the IM speed rate (numSpeeds per min, R 
never more than 0.5) even though a good model for numSpeeds (R=0.73) was derived. 
No phyiscal explanation for this has yet surfaced and the author believes this to be a 
mathematical artifact of the technique used and the relatively few data samples. 


All Distance-Based Operations 


PTP performance for all distance-based operations generates a good model (R=0.9999) 
but is an example of an instance of a model with no degrees of freedom left and thus not 
a very representative model. The number of leftover degrees of freedom for such a model 
is | based on 14 initial observations and a model with 9 terms. Forcing a simpler model 
may generate more degrees of freedom with a worse fit model. A reportable model can 
thus be generated by forcing the search to not include quadratic terms. 


The resulting model for PTP performance for all distance-based operations is (R=0.71): 


Generalized Linear regression model: 
achievedSpacingError PTP ~ [Linear formula with 7 terms in 5 predictors] 


Distribution = Normal 


Estimated Coefficients: 


Estimate SE tstat pValue 
(Intercept) 2.3439 0.60345 3.8842 0.011593 
traffic_path_JELVO.SUBDY1 2.3578 1.0754 2.1924 0.079851 
traffic_path_ZIRAN.SUBDY1 -0.65848 0.70623 -0.93239 0.39393 
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fo} 


traffic_path_final_appr -3.1473 0.80223 = 39232 -011146 


IMlength -0.051403 0.015419 -3.3336 0.020696 
windAlongTrackComponents_mean -0.15433 0.04205 -3.6702 0.014441 
windAlongTrackComponents_traffic_mean 0.154 0.04213 3.6553 0.014667 
traffic_speedProfileDev_mean_lead1 0.10289 0.031514 3.2647 0.022328 
IMlength:windAlongTrackComponents_traffic_mean -0.00060412 0.00014685 -4.114 0.0092282 


14 observations, 5 error degrees of freedom 


Matlab includes a utility to graphically depict and evaluate the model (plotSlice). 
The following figure examplifies this utility for the above model (distance-based PTP 
error). The utility is interactive and allows for exploration of the model. For each 
included regressor (columns), the user can select values either manually (by entering or 
selecting values), or graphically (by moving each vertical blue line). The model output 
for the selected values is shown on the left of the y axis, including confidence intervals. 
Green lines in each regressor areas show the estimated response (confidence bounds as 
red dotted lines) for that regressor throughout its envelope. 


The shape of the green lines represent a useful graphical way to estimate the contribution 
to each regressor on the response independent of scale (because the y axis is common). 


Thus, from Figure 76, one can quickly graphically infer that the true wind along track 
components have the larger impact on PTP error performance (larger slopes) than the 
path or IM length. The fact that ownship and traffic true wind component factor slopes 
have opposite signs may be an artifact of the model because it is created for all distance- 
based operations, which include a mix of same route operations (CAPTURE, 
MAINTAIN, SPACE) and merging route operations (CROSS). In same route operations 
the traffic true wind component should be similar to ownship’s. However in merging 
operations they may be much different. Thus a better model for distance-based PTP 
performance should be obtained by segregating operation types because the wind 
components look predominent but have different effects depending on the operation type. 
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Figure 76: plotSlice Output for PTP Error Model (Distance-Based Operations) 


A simple model for numSpeeds can be found for all distance-based operations with 
relatively good quality (R=0.79) and leftover degrees of freedom 


Generalized Linear regression model: 


log(numSpeeds) ~ 1 + IMlength + windAlongTrackComponents_ traffic _mean + 
traffic_speedProfileDev_mean_ leadl 


Estimated Coefficients: 


Estimate SE tStat pValue 
(Intercept) 1.1995 0.28519 4.2059 2.6006e-05 
IMlength 0.011589 0.0043131 2.6869 0.007211 
windAlongTrackComponents_traffic_mean -0.0057008 0.0029095 -1.9594 0.05007 
traffic_speedProfileDev_mean_ leadl -0.026081 0.015264 -1.7087 0.087503 


14 observations, 10 error degrees of freedom 


This model has three linear terms of generally the same scale and thus the estimates can 
be numerically compared for effect strength. Since IM length takes values up to 100nm 
but the wind components and speed errors take values between ~ -25 to +25 (half the 
range of IM length) all three components of the model have similar strength effects. 


A model for the IM Speed Rate can be derived (R=0.8) when constraining to linear terms 
to increase the number of leftover degrees of freedom: 


Generalized Linear regression model: 
reciprocal (numSpeeds_perMin)~1 + traffic_path + chain_positionSTR + traffic_speedProfileDev_mean_lead1 


Distribution = Gamma 


Estimated Coefficients: 


Estimate SE tStat pValue 
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(Intercept) 0.92623 0.10388 8.9165 1.9844e-05 


traffic_path_JELVO.SUBDY1 -3.0217 0.99707 -3.0305 0.016296 
traffic_path_ZIRAN.SUBDY1 1.0257 0.23235 4.8447 0.0012804 
traffic_path_final_appr -0.98787 0.24441 -4.0419 0.003726 
chain_positionSTR_2 3.6934 0.96292 34835) 0.0049778 
traffic_speedProfileDev_mean_lead1 0.0642 0.015903 4.0369 0.0037518 


14 observations, 8 error degrees of freedom 


This model does not include IM length as numSpeeds does (because it represents a rate 
per operational time, strongly correlated to IM length) and thus the term likely became 
insensitive. Interestingly, the terms retained include traffic path and chain position, which 
were not part of the model for numSpeeds. Again, this is difficult to explain and is 
currently attributed to the small sample size and mathematical artifact of the two different 
distributions imposed on the model as a result of the different nature of their responses 
(positive integers for numSpeeds, implying a Poisson distribution and log link funtion; 
positive continuous numbers for speed rate, implying a Gamma distribution and inverse 
link function). 


To obtain models for other IM performance metrics, we need to start building more 
specific models for each clearance types. 


Time-Based MAINTAIN Operations 


PTP error for time-based MAINTAIN operations results in a good model (R=0.8014) 
with only one interaction term. 


Generalized Linear regression model: 


achievedSpacingError PTP ~ 1 + hAtStart + IMlength + chain_positionSTR + ownship _path*own_aircraft 


Estimated Coefficients: 


Estimate SE tstat pValue 
(Intercept) -11.692 2.8247 -4.1392 0.0011644 
hAtStart 0.00042514 9.7666e-05 45353 0.00078266 
ownship path_JELVO.SUBDY1 4.4175 1.9912 2.2185 0.044942 
own_aircraft_B757 -5.3393 1.8976 -2.8137 0.014641 
IMlength -0.064419 0.025742 -2.5025 0.026466 
chain_positionSTR_1 2.834 1.381 2.0521 0.060867 
ownship path _JELVO.SUBDY1:own_aircraft_B757 8.5106 1.4832 56 f381L 6.8436e-05 


20 observations, 13 error degrees of freedom 


hAtStart is a surrogate for condition type (A/B/C). When we instead use condition type as 
a categorical variable in the model, the model is poorer and the term is absent. It appears 
to be a strong term in the model, even though technically the data is clustered mostly near 
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two values (23,000 ft and 35,000 ft). However, because several runs were truncated 
beyond the initial start, some values exist below 23,000 ft. Also, in some instances, 
aircraft were kept lower than FL230 on B runs. The 757 aircraft appears to decrease the 
quality of the PTP delivery and has an interaction term with the JELVO path. It can be 
concluded that the data reveals poorer MAINTAIN performance for the 757, especially 
on the en route path (it should be noted that ownship_path in this subset only takes two 
values [en route path is nominal], as no time-based MAINTAIN operations were flown 
with ownship on either UPBOB or ZIRAN.SUBDY). This corroborates the ANOVA 
results reported in the previous section, with this model now including strength of terms, 
with the ownship path being the strongest term, followed aircraft type and chain position. 


A good model for numSpeeds can be obtained for time-based MAINTAIN operations 
(R=0.76) 


Generalized Linear regression model: 


log(numSpeeds) ~ 1 + ownship_ path + IMlength + windAlongTrackComponents_ mean 


Estimated Coefficients: 


Estimate SE tStat pValue 
(Intercept) -0.20471 0.45035 -0.45455 0.64943 
ownship path _JELVO.SUBDY1 1 23D 0.41509 4.1512 3.3076e-05 
IMlength 0.010767 0.0040359 2.6679 0.0076333 
windAlongTrackComponents_mean -0.0095997 0.0036042 -2.6635 0.0077338 


20 observations, 16 error degrees of freedom 


This model has three terms of relatively equal strength, revealing better performance for 
the (nominal) en route path. 


The best model that can be obtained for IM speed rate has a R value of 0.61, not good 
enough for reporting. Several attempts were made at looking for better models, including 
replacing the hAtStart variable with the condition type (A/B/C), adding both, removing 
IM length, etc., to no avail. Again, the fact that a good model can be had for numSpeeds 
but not for IM Speed rate is attributed to the mathematical artifacts of the techniques 
used. 


A model for the spacingError RMS value of the maintain segment can be reported 
(R=0.7). 


Generalized Linear regression model: 
reciprocal (spacingErrorMaintainSegment_rms) ~ 1 + hAtStart + ownship path + IMlength + IMlength*2 


Distribution = Gamma 


Estimated Coefficients: 


Estimate SE tStat pValue 


(Intercept) 2.8341 0.55244 5.1302 8.3527e-05 
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hAtStart -2.8374e-05 6.3803e-06 -4.4471 0.00035364 


ownship path_JELVO.SUBDY1 -0.50912 0.13545 -3.7588 0.0015647 
IMlength -0.03745 0.012057 -3.1062 0.006419 
IMlength*2 0.000245 6.6417e-05 3.6888 0.0018216 


22 observations, 17 error degrees of freedom 


To obtain this model, the lead aircraft speed error (delay condition) had to be removed 
from the list of regressors to search. When including it, the resulting model does not 
include the term but excludes a lot more terms and is much poorer. The reported model 
includes the initial altitude variable as a surrogate for condition type. When including 
both, the model does not improve and includes both terms revealing the B condition 
(descent) as having slightly better maintain performance than the level (A) condition. 
However the simpler model is reported above, excluding the condition type term. IM 
length appears both as a linear and quadratic term. The presence of the quadratic term can 
be explained by the fact that the RMS metric is dependent on the length of the operation 
(better performance for longer operation) and is quadratic in nature (square root of a 
squared term). 


Time-Based CAPTURE Operations 


A model for PTP error for time-based CAPTURE operations can be reported (R=0.79) 
but is relatively more complex than reported models so far (nine terms with two 
interaction terms). 


Generalized Linear regression model: 
achievedSpacingError PTP ~ [Linear formula with 9 terms in 6 predictors] 


Distribution = Normal 


Estimated Coefficients: 


Estimate SE tstat pValue 
(Intercept) -6.5604 1.9018 -3.4495 0.0022843 
ownship path_JELVO.SUBDY1 2: 15716 1.7563 1.2285 0.23225 
ownship path_ZIRAN.SUBDY1 5.1184 1.8408 2.7806 0.010906 
own_aircraft_B757 2.3624 0.69007 3.4234 0.0024313 
IMlength 0.036663 0.019123 2.9272 0.068291 
windAlongTrackComponents_mean 0.0093661 0.015102 0.6202 0.5415 
chain_positionSTR_2 -4.4288 1.0596 -4.1798 0.00038888 
traffic_speedProfileDev_mean_lead1 -1.4018 0.26442 -5.3016 2.5462e-05 
IMlength:traffic_speedProfileDev_mean_leadl 0.013445 0.0026819 5.0132 5.1032e-05 
windAlongTrackComponents_mean:chain positionSTR_2 -0.17907 0.039066 -4.5837 0.00014499 


32 observations, 22 error degrees of freedom 


Graphical review of the model reveals that the term with the strongest effect is the true 
wind along track component, followed by path and chain position. The IMlength is a 
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weak component, likely helping delineate the capture from the maintain segment of the 
operation (longer IM lengths are likelier to have a longer maintain segment). The aircraft 
type effect has been seen in the ANOVA analysis, but that analysis also revealed a 
difference in performance across operation types, and yet neither hAtStart nor the 
condition type regressors are found in the resulting model. 


A model for tCapture can be derived (R=0.73) only when removing the lead delay 
condition regressor from the search, and including quadratic terms. Two quadratic terms 
are present: initialSpacingError and hAtStart. Thus, it would appear that the time to 
capture is strongly influenced by the altitude at the start of the operation (two terms) and 
initial spacing error (two terms). When attempting a model with condition type (A/B/C) 
instead of hAtStart, the resulting model is worse, indicating that altitude truly has an 
effect. The author believes this is a result of the effect of altitude on the estimated CAS 
decceleration segments that are derived and used by the CTD algorithm based on ground 
speed history, and transformed to airspeed using altitude and interpolated forecast winds 
at those altitudes. 


Generalized Linear regression model: 


reciprocal (tCapture) ~ 1 + initialSpacingError + hAtStart + chain_positionSTR + 
initialSpacingError’2 + hAtStart%2 


Distribution = Gamma 


Estimated Coefficients: 


Estimate SE tStat pValue 
(Intercept) -0.001392 0.0047546 -0.29277 0.77185 
initialSpacingError 5.1354e-06 4.5997e-06 1.1164 0.27372 
hAtStart 7.8547e-07 3.8461e-07 2.0422 0.05065 
chain_positionSTR_2 -0.0025772 0.00047388 -5.4385 8.3746e-06 
initialSpacingError%’%2 -1.3324e-06 2.2232e-07 -5. 9933 1.8633e-06 
hAtStart*2 -1.7001e-11 7.2086e-12 -2.3584 0.02557 


34 observations, 28 error degrees of freedom 


A model for IM speed rate can be derived (R=0.84) but is fairly complex (13 terms, 
including 2 quadratic terms). Attempts at deriving simpler models of equivalent quality 
have failed, thus the complex model is reported here. 


Generalized Linear regression model: 
numSpeeds perMin ~ [Linear formula with 13 terms in 7 predictors] 
Distribution = Gamma 


Estimated Coefficients: 


Estimate SE tstat pValue 
(Intercept) -4.1214 1.4968 =201535 0.013075 
initialSpacingError -0.0099237 0.0019638 =5* 0533 8.2745e-05 
hAtStart 0.00062461 0.00012863 4.8558 0.00012696 
ownship path _JELVO.SUBDY1 -1.4023 0.5469 -2.5641 0.019516 


REV A D780-10416-1 Page 103 of 116 


ownship path_ZIRAN.SUBDY1 -1.1872 0.57737 -2.0562 0.054558 


IMlength 0.013612 0.0048363 2.8145 0.011475 
windAlongTrackComponents_mean 0.083951 0.01444 5.8138 1.654e-05 
chain_positionSTR_2 -1.1778 0.17714 -6.6491 3.0663e-06 
traffic_speedProfileDev_mean_lead1 0.065325 0.010926 5.9791 1.1764e-05 
initialSpacingError:windAlongTrackComponents_ mean -0.00016399 6.1314e-05 -2.6746 0.015463 
hAtStart:windAlongTrackComponents_ mean -2.5841e-06 4.8408e-07 =5:.3382 4.4921e-05 
chain_positionSTR_2:traffic_speedProfileDev_mean_lead -0.17243 0.027696 -6.2258 7.1214e-06 
initialSpacingError%’2 0.00012426 4.6235e-05 2.6876 0.015041 
hAtStart*2 -1.376e-08 2.4345e-09 -5.6522 2.3154e-05 


32 observations, 18 error degrees of freedom 


Interestingly, in this instance, no good model for numSpeeds can be derived, exhibiting 
opposite output from the MAINTAIN regression models. This is likely a result of the 
more complex nature of the CAPTURE operation, including a capture and maintain 
segment, where the maintain segment is known to generate more speeds. Thus when 
looking at the IM speed rate, the number of speeds are averaged and the differentiated 
effect of the number of speeds between the capture and maintain segments is minimized. 


A model for the RMS error of the maintain segment can be found (R=0.95), but includes 
12 terms in 8 predictors, including 3 interaction terms. An initially better model was 
found with quadratic terms as well but was simplified at a small reduction in fit. It is 
expected that initial spacing error and IM length affect maintain performance, as well as 
the wind and lead delay condition. It is surprising to find terms related to the path and 
aircraft type because ANOVA analysis at the same alpha level revealed no statistical 
differences in those groups for this metric. 


Generalized Linear regression model: 
spacingErrorMaintainSegment_rms ~ [Linear formula with 12 terms in 8 predictors] 


Distribution = Gamma 


Estimated Coefficients: 


Estimate SE tstat pValue 
(Intercept) 1.7429 0.17861 9.758 1.3003e-08 
initialSpacingError 0.00029001 0.00014951 1.9397 0.068245 
hAtStart -5.1298e-05 6.7166e-06 -7.6375 4.7134e-07 
ownship path_JELVO.SUBDY1 -0.61875 0.070122 -8.8239 5.9049e-08 
ownship path_ZIRAN.SUBDY1 S02 52174 0.06294 -8.2889 1.4732e-07 
own_aircraft_B757 -0.11934 0.033053 -3.6106 0.0019995 
lead_aircraft_F900 0.11528 0.034054 3.3852 0.0032977 
lead_aircraft_B737 0.13542 0.042584 3.1802 0.0051831 
IMlength -0.0083808 0.0012343 -6.7899 2.3296e-06 
windAlongTrackComponents_mean -0.0034083 0.00032258 -10.566 3.8033e-09 
traffic_speedProfileDev_mean_lead1l 0.017474 0.0028012 6.238 6.9471e-06 
initialSpacingError:traffic_speedProfileDev_mean_leadl -8.8419e-05 2.8454e-05 -3.1074 


0.0060797 
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hAtStart:IMlength 4.8386e-07 6.0305e-08 8.0235 2.3504e-07 


own_aircraft_B757:traffic_speedProfileDev_mean_leadl -0.022233 0.0028524 -7.7944 3.5434e-07 


32 observations, 18 error degrees of freedom 


Time-Based CROSS Operations 


Models for CROSS operations include more regressors in the stepwise search. This is 
because we must account for the FIM pair to be on different paths, thus introducing the 
effect of traffic path and traffic wind components. Also, the ABP placement (NALTE and 
ZAVYO) and the DTG to the ABP at IM start are included in the search. hAtStart is 
included in the initial list of regressors even though only B scenarios were run for 
CROSS to determine whether altitude at start is an individual component of performance. 
As a result, the models reported are generally more complex than previous models. 


A model for PTP performance can be found although just below the reporting threshold 
(R=0.69). This model was manually arrived at after finding a better fit at the reporting 
threshold but with a term left with a pValue beyon 0.05 (DTG_ABP). Although the fit 
reduces to 0.69 after removal of the term, the model is shown anyway in this case. 


Generalized Linear regression model: 
achievedSpacingError PTP ~ [Linear formula with 8 terms in 7 predictors] 


Distribution = Normal 


Estimated Coefficients: 


Estimate SE tStat pValue 
(Intercept) 10.323 2.0295 5.0863 5.7506e-06 
ownship path_JELVO.SUBDY1 -3.5819 26:00:37 -1.7788 0.08148 
ownship_path_UPBOB1 10.546 23221 4.7485 1.8221e-05 
lead_aircraft_B737 3.1199 3.2125 0.97116 0.33624 
lead_aircraft_B757 10.63 4.5345 2.3442 0.023166 
ABP_NALTE -5.4998 1.7124 -3.2118 0.0023317 
chain_positionSTR_2 =13.4059 4.8407 -2.6978 0.0095452 
traffic_speedProfileDev_mean_lead1 0.50215 0.11725 4.2829 8.5751e-05 
DTG_ABP:windAlongTrackComponents traffic_mean 0.0042044 0.00065859 6.3839 5.9754e-08 
ABP NALTE: traffic _speedProfileDev_mean_leadl -0.39176 0.16745 -2.3396 0.023423 


59 observations, 49 error degrees of freedom 


This model again shows than when UPBOB is used, in large part with the 737 following 
the 757, the performance will be degraded by at least 7 s (10.546 s additional error for 
UPBOB, 10.63 s additional error for 757 lead and -13.059 s additional error for the 
second chain position). Initial Spacing Error and hAtStart are dropped terms. Only the 
traffic wind components are retained and included in the model when the ABP is at 
NALTE, suggesting the non-coincident route portion at altitude (prior to NALTE) has an 
effect. 
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A model for ABP performance can also be found at the reporting threshold (R=0.7) at 
about the same level of complexity. It should be noted that this is expected because the 
set of time-based CROSS operations includes 41 (out of 68) operations with ABP at 
ZAVYO, where the ABP performance is the same as the PTP performance. Independent 
models for ZAVYO and NALTE CROSS operations were sought and are reported later. 


Generalized Linear regression model: 
achievedSpacingError ABP ~ [Linear formula with 9 terms in 6 predictors] 


Distribution = Normal 


Estimated Coefficients: 


Estimate SE tstat pValue 
(Intercept) 15.819 6.6818 2.3674 0.021991 
hAtStart -0.0007307 0.00030278 -2.4133 0.019677 
ownship path_JELVO.SUBDY1 -1.4601 2.3894 -0.61107 0.54403 
ownship_path_UPBOB1 13.663 2.1114 6.47 4.7642e-08 
windAlongTrackComponents_traffic_mean -0.33685 0.15082 -2.2335 0.030209 
chain_positionSTR_2 -6.6658 2.4974 -2.669 0.010344 
traffic_speedProfileDev_mean_lead1 0.37932 0.099676 38055 0.00040084 
hAtStart:DTG_ABP 5.425e-06 1.7486e-06 3.1024 0.0032124 
DTG_ABP:windAlongTrackComponents_traffic_mean 0.01039 0.0023235 4.4715 4.7463e-05 

windAlongTrackComponents_traffic_mean:chain_positionSTR_2 -0.2144 0.098713 Se ae a 


0.034831 


58 observations, 48 error degrees of freedom 


No model for numSpeed can be reported for time-based CROSS operations (best model 
is R=0.64), but a model for IM Speed Rate is just above the reporting threshold (R=0.71). 
This model has only five predictors but includes two quadratic and two interaction 
effects. ABP placement is not retained in the model even though the ANOVA analysis 
revealed a statistically significant difference among the NALTE vs ZAVYO ABP 
placement scenarios. 


Generalized Linear regression model: 
numSpeeds perMin ~ [Linear formula with 9 terms in 5 predictors] 


Distribution = Gamma 


Estimated Coefficients: 


Estimate SE tstat pValue 
(Intercept) 0.85872 0.25063 3.4262 0.0012476 
ownship path_JELVO.SUBDY1 0.60095 0.16892 335577 0.00084191 
ownship_path_UPBOB1 0.89083 0.28296 3.1483 0.0027952 
DTG ABP 0.019597 0.0040067 4.8909 1.1234e-05 
windAlongTrackComponents_mean -0.05254 0.011331 -4.6367 2.6554e-05 
chain_positionSTR_2 0.74213 0.38219 1.9418 0.057925 
DTG_ABP:windAlongTrackComponents_mean 0.00043781 0.00015934 2.7477 0.0083761 


REV A D780-10416-1 Page 106 of 116 


DTG_ABP:chain_positionSTR_2 -0.016455 0.0072127 -2.2814 0.026905 
windAlongTrackComponents_mean*2 -0.0012137 0.0001978 -6.1362 1.4431e-07 


traffic_speedProfileDev_mean_lead1%*2 -0.0025499 0.00065815 -3.8744 0.00031743 


59 observations, 49 error degrees of freedom 


A submodel for ABP performance for the time-based CROSS operations with ABP at 
NALTE is better than the general model (R=0.78). The model is simpler as well (four 
predictors) but with both degrees (linear and quadratic) of the chain lead delay profile 
effects. The model reveals that the performance is worse when the traffic aircraft is on the 
ZIRAN transition. Interestingly, DTG_ABP is not retained but IMlength acts as a 
surrogate. This is strange because, for all NALTE operations, the maintain segment is 
fixed in length equal to the along track distance from NALTE to ZAVYO on the common 
segment. 


Generalized Linear regression model: 
achievedSpacingError ABP ~ [Linear formula with 6 terms in 4 predictors] 


Distribution = Normal 


Estimated Coefficients: 


Estimate SE tstat pValue 
(Intercept) 165.77 35.461 4.6748 0.00029923 
hAtStart -0.0079594 0.001667 -4.7748 0.00024575 
traffic_path_ZIRAN.SUBDY1 6.565 1.4777 4.4428 0.00047453 
IMlength 0.13269 0.053818 2.4656 0.026226 
traffic_speedProfileDev_mean_lead1 0.26685 0.0807 3.3067 0.0047928 
traffic_speedProfileDev_mean_lead1%*2 0.026087 0.0071128 3.6676 0.0022859 


21 observations, 15 error degrees of freedom 


No other models for NALTE operations have been found at or above the reporting 
threshold. The maintain segment performance (R=0.68) and the IM speed rate (R=0.58) 
come the closest but are not fitting well enough to inspect. 


The ZAVYO only PTP performance model for time-based CROSS is better than the 
general model (R=0.72) 


Generalized Linear regression model: 
achievedSpacingError PTP ~ [Linear formula with 10 terms in 7 predictors] 


Distribution = Normal 


Estimated Coefficients: 


Estimate SE tStat pValue 
(Intercept) 86.388 20.38 4.2389 0.00025012 
initialSpacingError 0.093816 0.02357 3.9803 0.00049219 
hAtStart 0.0021475 0.00072167 2.9757 0.0062421 
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ownship path_JELVO.SUBDY1 -8.2759 4.2296 -1.9567 0.061207 


ownship_ path_UPBOB1 9.9294 3.6162 2.7458 0.010806 
traffic_path_ZIRAN.SUBDY1 6.73 3.8951 1.7278 0.095887 
IMlength $34.3773 0.79199 -4.2643 0.00023399 
windAlongTrackComponents_mean -1.0064 0.53438 -1.8833 0.070898 
traffic_speedProfileDev_mean_lead1 0.74927 0.13713 5.4641 9.9158e-06 
hAtStart:windAlongTrackComponents_ mean 6.5203e-05 2.518e-05 2.5895 0.015541 
IMlength*2 0.02311 0.0052582 4.3949 0.00016595 


37 observations, 26 error degrees of freedom 


No other models for ZAVYO operations have been found at or above the reporting 
threshold. 


Distance-Based CROSS Operations 


MAINTAIN, CAPTURE, and SPACE operations each have less than five operations 
flown in distance and thus are not inspected for regression modeling. CROSS operations 
have seven operations conducted in distance. 


A simple model for PTP spacing can be found with enough error degrees of freedom left 
over to be of some significance (R=0.8652). The model has two terms: the wind effect 
(for ownship only) and the chain lead delay condition. Initial spacing error and IM length 
are dropped as terms as well as all geometric components (path and hAtStart). 


Generalized Linear regression model: 
achievedSpacingError PTP ~ 1 + windAlongTrackComponents_ mean + traffic_speedProfileDev_mean_leadl 


Distribution = Normal 


Estimated Coefficients: 


Estimate SE tStat pValue 
(Intercept) -0.38386 0.18026 -2.1294 0.10027 
windAlongTrackComponents_mean -0.035632 0.0056817 -6.2714 0.0032995 
traffic_speedProfileDev_mean_lead1 0.057501 0.019673 2.9228 0.043123 


7 observations, 4 error degrees of freedom 


All distance-based CROSS operations were performed with ABP at ZAVYO and thus 
ABP performance should be no different than PTP performance and thus no independent 
regression model for ABP performance is sought. By the same token, no regression 
model for maintain RMS is sought because there was no maintain segment in those 
operations. 


The models derived for numSpeeds and IM Speed Rate are simple and of good fit, but 
have only 2 degrees of freedom left over and thus are not reported. 
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Summary of Regression Analysis 


Regression models for key IM performance metrics as functions of IM operational input 
type regressors were sought. Sixteen models have been derived with a fit better than 0.7 
when using the adjusted R squared metric for fit quality. Most models retain between 
three and five predictors in linear, interaction, and quadratic terms. The simplest model 
has two predictors and the most complex model has eight predictors. The best fit model is 
the most complex (R=0.95). A summary table of the models attempted and those reported 
is in Figure 77. 


AllTime Based All Distance Based Time Based Time Based Time Based Distance Based 


Opearations Operations MAINTAIN CAPTURE CROSS CROSS 
R=0.71, 5 R=0.8, 5 R=0.79, 6 R=0.69, 7 R=0.87, 2 
PTP Performance No Model Found predictors predictors predictors predictors predictors 
R=0.7, 6 
ABP Performance N/A N/A N/A N/A predictors N/A 
R=0.79, 3 R=0.76, 3 No Model No Model No Model 
numSpeeds R=0.73, 7 predictors predictors predictors Found Found Found 
No Model R=0.84, 7 0.71, 5 No Model 
IMspeedRate No Model Found R=0.8, 3 predictors Found predictors predictors Found 
t=0.7, 3 R=0.73, 3 
tCapture N/A N/A predictors predictors N/A N/A 
R=0.7, 3 R=0.95, 8 No Model 
maintainSegment_RMS N/A N/A predictors predictors Found N/A 


Figure 77: Regression Model Search Summary 


In general, the initial spacing error is not present in the delivery performance models, 
indicating that the algorithms implemented have enough control authority to deal with the 
conditions they faced during flight test with no impact on performance. Delivery 
performance is affected more by wind conditions and IM length, as well as path (as 
ANOVA analysis demonstrated). Capture performance is influenced more by the initial 
spacing error as is expected. The overall takeaway from this exercise is the extent of the 
influence of the path (and thus its associated design) on all aspects of performance. 
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7. Summary and Conclusions 


The ATD-1 flight test program resulted in a body of 144 FIM operations to be analyzed 
for performance. In general, the observed performance is within or slightly outside of the 
MOPS performance criteria for a wide variation of input conditions designed to cover the 
spectrum of realistic operations in a typical airspace. The major performance criteria 
inspected include 


e PTP delivery. 

e ABP delivery (CROSS operations only). 

e Maintain Segment Error. 

e Capture time and Capture Rate (CAPTURE operations only). 
e Number of IM speeds, and IM speed rate. 


When taken in aggregate, time-based operations delivered at the PTP slightly worse than 
the MOPS criterion of 10 s of error at 95%. The empirical probability of an error less 
than 10 s is 88.3%. This is driven mostly by CROSS operations which, in aggregate, have 
a probability of delivering less than 10 s of error at the PTP of 78%, whereas no instances 
of errors greater than 10 s were seen for MAINTAIN, SPACE, or CAPTURE operations. 


The slightly worse PTP performance of CROSS operations is attributed to instances with 
a late merge including the UPBOB route for the trail aircraft. Executions of CROSS 
operations with an upstream merge have significantly better performance. It is believed 
that a different design for the UPBOB1 STAR, with a better-matched nominal altitude 
and speed profile, would improve performance. A different design for the UPBOB1 
STAR altitude and speed profile was drafted and a few operations were performed to 
enable comparisons that will be carried out by NASA at a later date. 


As a general conclusion from the observed data, it would seem that better PTP 
performance for CROSS operations should be sought operationally by placing the ABP 
upstream of the PTP at or downstream of route merges, thereby creating a maintain 
segment. Route design with late merges near the PTP are not conducive to good PTP 
delivery performance. 


Similar effects are observed at the ABP. Overall, time-based CROSS operations have an 
empirical probability of ABP errors of less than 10 s of only 69%. However, operations 
with the ABP at NALTE have 78% observed errors below 10 s, and when excluding a 
number of conditions where the operation started with insufficient control authority 
(distance to ABP too short for the initial spacing error), that probability increases to 88%. 
It is believed that, with better design of the nominal altitude and speed profile at NALTE 
for both the ZIRAN and JELVO transitions of the SUBDY1 STAR, the ABP 
performance would meet the MOPS criterion. 
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Maintain segment performance for MAINTAIN, CAPTURE and CROSS (NALTE) 
operations has a mean of 3.6 s using the RMS error metric. Although the MOPS does not 
define maintain performance in this way, it is believed that, in aggregate, the flight test 
demonstrated acceptable maintain performance. 


Thirty of the 34 (88%) CAPTURE operations had capture rates better than the MOPS 
criteria of 3 s or error per minute of operation. The majority of the cases that did not meet 
this level of performance are attributed to extreme cases at the back of the chain with 
medium to large delay at the front of the chain. 


This level of overall performance is achieved with an average rate of 0.57 IM speeds per 
minute (one IM speed command every two minutes). Pilots have nominally indicated in 
surveys that the number of IM speeds was acceptable, but it has been observed that 
MAINTAIN operations tend to issue more speeds than other clearances, as well as 
include more reversions. This has been attributed to the design of the CTD algorithm and 
the lack of hysteresis in cases where the speed error is near the discretized speed action 
threshold. 


This performance is achieved (1) through a wide variety of initial spacing errors (96 s too 
close to 84 s too late); (2) on three different routes; (3) for descent, cruise, and final 
approach operations; (4) two different aircraft; and (5) 19 different days of experienced 
wind conditions (from 61 kt of average headwind to 67 kt of average tailwind) and 
associated forecasts and errors (from 2 kt to 19 kt of average RMS along-track forecast 
error). 


The Fuel Burn analysis performed compared only fuel burn across operations in the flight 
test and have no comparative basis to a non-FIM OPD operation. In general, the analysis 
revealed expected differences between aircraft types and correlated to the number of FIM 
speeds issued; the more speeds issued, the more fuel burned. 


The PDE analysis performed revealed overall small values for PDE, including less than 
2000 ft average vertical difference among the descent paths between the FMS and FIM 
computed vertical trajectories. This is an excellent result given the difference in fidelity 
of the two system trajectory generators. 


The along track wind forecast error analysis revealed a large variation of forecast errors 
throughout the 19 flight days. In general, the forecast error tended to go through a large 
variation near the RF turns on the SUBDY procedure. This is expected because the 
forecasts were generated from a column of air over the Grant County airport for the low 
altitude forecast breakpoints (surface, 6,000 ft, 12,000 ft) and the trajectory geometry 
changes direction significantly through the RF turns in the 6,000 ft to 12,000 ft altitude 
band. 


Overall, the Flight Test effort was a success, delivering data for realistic FIM operations 
on the first prototype of a FIM system that revealed a majority of the operations to be 
within MOPS standards. The prototype is based on an implementation of the NASA 
ASTAR algorithm with additions to fit the demand of the flight test, namely a distance- 
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based adaptation, an implementation of final approach spacing, and a Mach/CAS 
transition implementation. The analysis performed reveals that minor changes and 
finalization of the control algorithms should lead to system performance that meets or 
exceeds the MOPS standards. In particular, improvements in the CTD algorithm during 
maintain operations should lead to fewer issued IM speeds with no loss of performance. 


The flight test also revealed the impact of route design on FIM performance, in particular 
the altitude and speed profile information embedded within the design in the form of 
altitude and speed constraints at waypoints. Given that the route design establishes the 
baseline profile for the FIM system, a design that seeks smoother nominal speed 
transitions along the route would lead to less drastic speed changes during IM operations. 
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